# Multimodal fusion models


📂 model-architectures

## MODEL ARCHITECTURES

# Multimodal fusion models

Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.

## MENTAL MODEL

“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.

## DATA FLOW

- Text / image / audio / video

- Modality encoders or tokenizers

- Projector, cross-attention, or shared backbone

- Joint representation / language decoder

- Text, media, or action output

## How it trains

Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.

## How inference runs

Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.

## Strengths

- •
Grounds language in visual or acoustic inputs

- •
Supports document, chart, image, video, and audio workflows

- •
Can reuse strong pretrained modality components

## Trade-offs

- •
Modality imbalance and connector bottlenecks

- •
Media tokens consume substantial context and compute

- •
Evaluation must separate perception, grounding, reasoning, and generation

## Use it when

- The task genuinely depends on non-text evidence

- Modality-specific slices and abstention are evaluated

- The architecture exposes enough detail for cost and privacy review

## Avoid or challenge it when

- A transcript, OCR result, or structured extractor is sufficient

- “Supports images” is assumed to mean precise perception

- Sensitive media handling and retention are undefined

## Illustrative published families

- • Flamingo cross-attention architecture

- • BLIP-2 learned querying connector

- • Shared-token multimodal decoder systems

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures