# Audio, speech & music models


📂 model-architectures

## MODEL ARCHITECTURES

# Audio, speech & music models

A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.

## MENTAL MODEL

Choose the task first. ASR maps audio to text; TTS and music generation map conditions to acoustic or codec representations; understanding maps audio to labels or embeddings.

## DATA FLOW

- Waveform or spectrogram

- Acoustic / codec representation

- Encoder, seq2seq, diffusion, or token generator

- Text, labels, codec tokens, or waveform

- Task-specific decoding

## How it trains

Objectives vary: transcription uses sequence likelihood, representation learning may mask audio, codec generators predict discrete audio tokens, and diffusion systems learn denoising in waveform, spectrogram, or latent space.

## How inference runs

ASR usually encodes audio then decodes text; generation may predict codec tokens autoregressively or iteratively denoise. Streaming constraints can require chunked encoders and bounded look-ahead.

## Strengths

- •
Speech transcription and translation

- •
Natural speech, sound, and music synthesis

- •
Audio embeddings support search, moderation, and classification

## Trade-offs

- •
Accents, noise, overlap, code-switching, and domain vocabulary shift quality

- •
Long-form generation needs temporal structure and consistency

- •
Voice identity, consent, watermarking, and copyright need explicit governance

## Use it when

- The modality itself carries needed information

- You have task-, language-, and environment-specific evaluations

- Latency and streaming requirements are designed up front

## Avoid or challenge it when

- A transcript already contains all relevant information

- Identity cloning lacks informed consent

- One benchmark is being used as proof across languages and acoustic conditions

## Illustrative published families

- • Whisper-style encoder-decoder ASR

- • AudioLM-style hierarchical codec-token generation

- • Waveform or latent diffusion audio systems

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures