# Autoregressive visual & audio models


📂 model-architectures

## MODEL ARCHITECTURES

# Autoregressive visual & audio models

Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.

## MENTAL MODEL

Make media look like a language: tokenize it, choose an ordering, predict the next code, then decode.

## DATA FLOW

- Media tokenizer / codec

- Ordered discrete codes

- Causal token model

- Generated code sequence

- Media decoder

## How it trains

A learned codec is trained for reconstruction; the prior minimizes next-code cross-entropy, optionally conditioned on text, prior frames, semantic tokens, or coarse-scale codes.

## How inference runs

Codes are generated serially or in hierarchical/blockwise schedules, then decoded. Ordering determines latency and which dependencies are easy to model.

## Strengths

- •
Unified token-based modeling across modalities

- •
Compatible with causal language-model infrastructure

- •
Exact discrete likelihood for the chosen tokenization

## Trade-offs

- •
Long code sequences create serial latency

- •
Tokenizer artifacts bound output quality

- •
A one-dimensional ordering can be awkward for spatial or multiscale structure

## Use it when

- A strong discrete codec exists

- Cross-modal token modeling or continuation is important

- Hierarchical generation can meet latency goals

## Avoid or challenge it when

- Long serial decoding misses the budget

- The codec loses critical perceptual detail

- A continuous diffusion representation is substantially simpler

## Illustrative published families

- • ImageGPT-style pixel/token modeling

- • VQ-token image generators

- • AudioLM-style codec-token hierarchy

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures