Generative Model Architectures

📂 model-architectures

MODEL ARCHITECTURES#

Model architecture field guide

Generative Model Architectures

Understand what actually sits behind text, embedding, image, video, audio, and multimodal products, then choose a backbone, representation, generator, and system wrapper deliberately.

22 architecture families · 5 groups · one page per family

THE ESSENTIAL DISTINCTION

A capability is not an architecture#

“Chat,” “image,” and “multimodal” describe an interface. Decoder-only, state-space, DiT, and MoE describe internal computation. RAG and tool use describe the runtime system around a model. One production system can legitimately carry all three kinds of label.

Write an architecture passport#

• 1 Capability: Inputs, outputs, user job, and required evidence

• 2 Representation: Tokens, vectors, pixels, codec codes, or continuous latents

• 3 Backbone: Encoder, decoder, seq2seq, attention, SSM, or hybrid

• 4 Capacity: Dense or expert-routed; total versus active parameters

• 5 Generation: Autoregressive, diffusion/flow, adversarial, or deterministic

• 6 System: Retrieval, tools, validators, permissions, monitoring, and humans

Browse all 22 families#

Each family has its own page with training, inference, strengths, trade-offs, and selection guidance.

PRODUCT CAPABILITIES#

What the product accepts and returns. These are deployment shapes, not mutually exclusive neural architectures.

Text & chat language models

Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.

Embedding models

Map text, images, audio, users, or items into vectors whose geometry is trained to preserve a useful notion of similarity.

Image generation & editing

Synthesize or transform pixels, most often through a text or image conditioner, a latent generator, and an image decoder.

Video generation

Generate spatial and temporal structure together using frame, patch, or latent video representations with cross-frame computation.

Audio, speech & music models

A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.

CORE BACKBONES#

How information moves through the trainable network. A deployed model may combine several of these choices.

Encoder-only Transformers

Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.

Decoder-only Transformers

Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.

Encoder–decoder Transformers

Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.

Mixture of Experts (MoE)

Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.

State-space & recurrent models

Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.

REPRESENTATION & GENERATION#

How images, audio, video, and other high-dimensional outputs are represented, learned, and sampled.

Autoencoders, VAEs & learned tokenizers

Learn an encoder that compresses data and a decoder that reconstructs it; variational and quantized variants shape the latent space for sampling or downstream generation.

Generative Adversarial Networks (GANs)

Train a generator to fool a discriminator while the discriminator learns to distinguish generated samples from training data.

Normalizing flows

Transform a simple distribution into a complex data distribution through a sequence of invertible mappings with tractable Jacobians.

Diffusion & score-based generation

Learn to reverse a gradual noising process, producing data by repeatedly transforming noise toward a sample.

Latent diffusion

Run diffusion in an autoencoder’s lower-dimensional latent space, then decode the generated representation back to pixels or another signal.

Diffusion Transformers (DiT)

Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.

Autoregressive visual & audio models

Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.

MODALITY INTERFACES#

How vision, text, audio, and other modalities are encoded, aligned, fused, or connected to a generator.

Vision encoders

Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.

Contrastive & dual encoders

Encode two inputs independently (such as a query and document or image and caption) and train matching pairs to land near each other.

Multimodal fusion models

Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.

AI SYSTEM ARCHITECTURES#

Runtime structures around a model that add knowledge, tools, control, and verifiable behavior.

Retrieval-augmented & hybrid systems

Retrieve evidence at request time and provide it to a generator, classifier, or agent instead of relying only on model parameters.

Tool-using & reasoning systems

Place a model inside a controller that can plan, call typed tools, inspect results, revise, verify, stop, and hand control to a person.

Model Architectures#

Model Architectures#