audio-speech-music-models
Audio, speech & music models
📂 model-architectures
MODEL ARCHITECTURES#
Audio, speech & music models
A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.
MENTAL MODEL#
Choose the task first. ASR maps audio to text; TTS and music generation map conditions to acoustic or codec representations; understanding maps audio to labels or embeddings.
DATA FLOW#
-
Waveform or spectrogram
-
Acoustic / codec representation
-
Encoder, seq2seq, diffusion, or token generator
-
Text, labels, codec tokens, or waveform
-
Task-specific decoding
How it trains#
Objectives vary: transcription uses sequence likelihood, representation learning may mask audio, codec generators predict discrete audio tokens, and diffusion systems learn denoising in waveform, spectrogram, or latent space.
How inference runs#
ASR usually encodes audio then decodes text; generation may predict codec tokens autoregressively or iteratively denoise. Streaming constraints can require chunked encoders and bounded look-ahead.
Strengths#
-
• Speech transcription and translation
-
• Natural speech, sound, and music synthesis
-
• Audio embeddings support search, moderation, and classification
Trade-offs#
-
• Accents, noise, overlap, code-switching, and domain vocabulary shift quality
-
• Long-form generation needs temporal structure and consistency
-
• Voice identity, consent, watermarking, and copyright need explicit governance
Use it when#
-
The modality itself carries needed information
-
You have task-, language-, and environment-specific evaluations
-
Latency and streaming requirements are designed up front
Avoid or challenge it when#
-
A transcript already contains all relevant information
-
Identity cloning lacks informed consent
-
One benchmark is being used as proof across languages and acoustic conditions
Illustrative published families#
-
• Whisper-style encoder-decoder ASR
-
• AudioLM-style hierarchical codec-token generation
-
• Waveform or latent diffusion audio systems