encoderdecoder-transformers
Encoder–decoder Transformers
📂 model-architectures
MODEL ARCHITECTURES#
Encoder–decoder Transformers
Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.
MENTAL MODEL#
A reader and a writer connected by cross-attention.
DATA FLOW#
-
Source sequence
-
Bidirectional encoder
-
Source memory
-
Causal decoder + cross-attention
-
Target sequence
How it trains#
Teacher-forced target-token likelihood is common. Text-to-text denoising or span corruption lets many tasks share the same input/output interface.
How inference runs#
The source is encoded once; the decoder generates the target autoregressively while attending to stable source states. Beam search is useful in some bounded tasks but is not universally best.
Strengths#
-
• Clear separation between source understanding and target generation
-
• Strong fit for translation, summarization, transcription, and transformation
-
• Source memory is reused throughout decoding
Trade-offs#
-
• Two stacks can increase parameters and deployment complexity
-
• Target decoding remains serial
-
• May be less convenient than one decoder-only interface for heterogeneous chat tasks
Use it when#
-
Output is tightly conditioned on a distinct input
-
Source and target have different structures or modalities
-
Faithful transformation matters more than open-ended continuation
Avoid or challenge it when#
-
A single general chat interface is the overriding requirement
-
No distinct source sequence exists
-
A non-generative encoder head is sufficient
Illustrative published families#
-
• Original Transformer
-
• T5
-
• Whisper speech recognition architecture