Autoregressive visual & audio models

📂 model-architectures

MODEL ARCHITECTURES#

Autoregressive visual & audio models

Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.

MENTAL MODEL#

Make media look like a language: tokenize it, choose an ordering, predict the next code, then decode.

DATA FLOW#

• Media tokenizer / codec

• Ordered discrete codes

• Causal token model

• Generated code sequence

• Media decoder

How it trains#

A learned codec is trained for reconstruction; the prior minimizes next-code cross-entropy, optionally conditioned on text, prior frames, semantic tokens, or coarse-scale codes.

How inference runs#

Codes are generated serially or in hierarchical/blockwise schedules, then decoded. Ordering determines latency and which dependencies are easy to model.

Strengths#

• • Unified token-based modeling across modalities

• • Compatible with causal language-model infrastructure

• • Exact discrete likelihood for the chosen tokenization

Trade-offs#

• • Long code sequences create serial latency

• • Tokenizer artifacts bound output quality

• • A one-dimensional ordering can be awkward for spatial or multiscale structure

Use it when#

• A strong discrete codec exists

• Cross-modal token modeling or continuation is important

• Hierarchical generation can meet latency goals

Avoid or challenge it when#

• Long serial decoding misses the budget

• The codec loses critical perceptual detail

• A continuous diffusion representation is substantially simpler

Illustrative published families#

• • ImageGPT-style pixel/token modeling

• • VQ-token image generators

• • AudioLM-style codec-token hierarchy

Commonly combines with#

PRIMARY REFERENCES#

Model Architectures#

Model Architectures#