Diffusion Transformers (DiT)

📂 model-architectures

MODEL ARCHITECTURES#

Diffusion Transformers (DiT)

Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.

MENTAL MODEL#

DiT changes the denoiser backbone; it does not eliminate the diffusion process, latent codec, conditioning, or sampler.

DATA FLOW#

• Noisy latent

• Patchify + position / time conditioning

• Transformer blocks

• Predicted latent update

• Sampler step → repeat

How it trains#

The model learns the same class of denoising, velocity, or flow objective as other diffusion systems while Transformer blocks mix information between latent patches.

How inference runs#

A sampler repeatedly invokes the Transformer at different noise or flow times, then a decoder maps the final latent to media.

Strengths#

• • Transformer scaling and implementation ecosystem

• • Global interactions between latent patches

• • Natural path to shared multimodal token processing

Trade-offs#

• • Attention cost grows with latent token count

• • Still requires repeated denoiser calls

• • “DiT” alone does not specify codec, objective, conditioning, or sampler

Use it when#

• Transformer scaling infrastructure is available

• Latent patch count and sampling latency fit the budget

• You need a flexible backbone for image or video generation

Avoid or challenge it when#

• A small U-Net already meets the objective

• The label is being used as a complete architecture specification

• High-resolution token counts overwhelm the target runtime

Illustrative published families#

• • DiT research family

• • Transformer denoisers inside latent image and video pipelines

Commonly combines with#

PRIMARY REFERENCES#

Model Architectures#

Model Architectures#