# Encoder–decoder Transformers


📂 model-architectures

## MODEL ARCHITECTURES

# Encoder–decoder Transformers

Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.

## MENTAL MODEL

A reader and a writer connected by cross-attention.

## DATA FLOW

- Source sequence

- Bidirectional encoder

- Source memory

- Causal decoder + cross-attention

- Target sequence

## How it trains

Teacher-forced target-token likelihood is common. Text-to-text denoising or span corruption lets many tasks share the same input/output interface.

## How inference runs

The source is encoded once; the decoder generates the target autoregressively while attending to stable source states. Beam search is useful in some bounded tasks but is not universally best.

## Strengths

- •
Clear separation between source understanding and target generation

- •
Strong fit for translation, summarization, transcription, and transformation

- •
Source memory is reused throughout decoding

## Trade-offs

- •
Two stacks can increase parameters and deployment complexity

- •
Target decoding remains serial

- •
May be less convenient than one decoder-only interface for heterogeneous chat tasks

## Use it when

- Output is tightly conditioned on a distinct input

- Source and target have different structures or modalities

- Faithful transformation matters more than open-ended continuation

## Avoid or challenge it when

- A single general chat interface is the overriding requirement

- No distinct source sequence exists

- A non-generative encoder head is sufficient

## Illustrative published families

- • Original Transformer

- • T5

- • Whisper speech recognition architecture

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures