model-architectures
Generative Model Architectures
📂 model-architectures
MODEL ARCHITECTURES#
Model architecture field guide
Generative Model Architectures
Understand what actually sits behind text, embedding, image, video, audio, and multimodal products, then choose a backbone, representation, generator, and system wrapper deliberately.
22 architecture families · 5 groups · one page per family
THE ESSENTIAL DISTINCTION
A capability is not an architecture#
“Chat,” “image,” and “multimodal” describe an interface. Decoder-only, state-space, DiT, and MoE describe internal computation. RAG and tool use describe the runtime system around a model. One production system can legitimately carry all three kinds of label.
Write an architecture passport#
-
1 Capability: Inputs, outputs, user job, and required evidence
-
2 Representation: Tokens, vectors, pixels, codec codes, or continuous latents
-
3 Backbone: Encoder, decoder, seq2seq, attention, SSM, or hybrid
-
4 Capacity: Dense or expert-routed; total versus active parameters
-
5 Generation: Autoregressive, diffusion/flow, adversarial, or deterministic
-
6 System: Retrieval, tools, validators, permissions, monitoring, and humans
Browse all 22 families#
Each family has its own page with training, inference, strengths, trade-offs, and selection guidance.
PRODUCT CAPABILITIES#
What the product accepts and returns. These are deployment shapes, not mutually exclusive neural architectures.
Text & chat language models
Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.
Embedding models
Map text, images, audio, users, or items into vectors whose geometry is trained to preserve a useful notion of similarity.
Image generation & editing
Synthesize or transform pixels, most often through a text or image conditioner, a latent generator, and an image decoder.
Video generation
Generate spatial and temporal structure together using frame, patch, or latent video representations with cross-frame computation.
Audio, speech & music models
A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.
CORE BACKBONES#
How information moves through the trainable network. A deployed model may combine several of these choices.
Encoder-only Transformers
Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.
Decoder-only Transformers
Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.
Encoder–decoder Transformers
Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.
Mixture of Experts (MoE)
Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
REPRESENTATION & GENERATION#
How images, audio, video, and other high-dimensional outputs are represented, learned, and sampled.
Autoencoders, VAEs & learned tokenizers
Learn an encoder that compresses data and a decoder that reconstructs it; variational and quantized variants shape the latent space for sampling or downstream generation.
Generative Adversarial Networks (GANs)
Train a generator to fool a discriminator while the discriminator learns to distinguish generated samples from training data.
Normalizing flows
Transform a simple distribution into a complex data distribution through a sequence of invertible mappings with tractable Jacobians.
Diffusion & score-based generation
Learn to reverse a gradual noising process, producing data by repeatedly transforming noise toward a sample.
Latent diffusion
Run diffusion in an autoencoder’s lower-dimensional latent space, then decode the generated representation back to pixels or another signal.
Diffusion Transformers (DiT)
Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.
Autoregressive visual & audio models
Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.
MODALITY INTERFACES#
How vision, text, audio, and other modalities are encoded, aligned, fused, or connected to a generator.
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
Contrastive & dual encoders
Encode two inputs independently (such as a query and document or image and caption) and train matching pairs to land near each other.
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
AI SYSTEM ARCHITECTURES#
Runtime structures around a model that add knowledge, tools, control, and verifiable behavior.
Retrieval-augmented & hybrid systems
Retrieve evidence at request time and provide it to a generator, classifier, or agent instead of relying only on model parameters.
Tool-using & reasoning systems
Place a model inside a controller that can plan, call typed tools, inspect results, revise, verify, stop, and hand control to a person.