latent-diffusion
Latent diffusion
📂 model-architectures
MODEL ARCHITECTURES#
Latent diffusion
Run diffusion in an autoencoder’s lower-dimensional latent space, then decode the generated representation back to pixels or another signal.
MENTAL MODEL#
Denoise a compact learned sketch instead of every raw pixel.
DATA FLOW#
-
Training media
-
Frozen or jointly trained autoencoder latent
-
Conditional latent denoiser
-
Sampled clean latent
-
Decoder → media
How it trains#
An autoencoder first learns a perceptual compression. The diffusion model then learns the denoising objective over those latents, often conditioned through cross-attention.
How inference runs#
Sample in the smaller latent tensor over several steps and decode once. Image-to-image and inpainting can begin from encoded and selectively noised source content.
Strengths#
-
• Lower denoising cost than raw high-resolution pixels
-
• Modular text conditioning and editing
-
• Reusable autoencoder and generator components
Trade-offs#
-
• The autoencoder creates a reconstruction ceiling
-
• Fine text or high-frequency detail can be lost in compression
-
• Two-stage failures are harder to attribute
Use it when#
-
High-resolution media generation must fit practical compute
-
Editing and conditional control are required
-
Codec reconstruction is validated on the target domain
Avoid or challenge it when#
-
The autoencoder drops task-critical details
-
End-to-end simplicity matters more than generation cost
-
Latent-space shortcuts could hide safety-relevant content
Illustrative published families#
-
• Latent Diffusion Models
-
• Stable Diffusion research lineage