# Vision encoders


📂 model-architectures

## MODEL ARCHITECTURES

# Vision encoders

Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.

## MENTAL MODEL

A visual reader. It creates representations; generation requires a decoder or generative model.

## DATA FLOW

- Image or frames

- Patch, convolutional, or hybrid stem

- Vision backbone

- Spatial tokens / feature pyramid / pooled vector

- Task head or modality connector

## How it trains

Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.

## How inference runs

The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.

## Strengths

- •
Reusable visual features

- •
Efficient classification, retrieval, and perception

- •
Can connect images to language models through a projector or cross-attention

## Trade-offs

- •
Resolution and patch size determine lost detail and token cost

- •
Training objectives create different blind spots

- •
A pooled vector is insufficient for many spatial tasks

## Use it when

- The system must understand or retrieve visual content

- You can choose pooled versus spatial features based on the task

- Small text, charts, and domain imagery are evaluated explicitly

## Avoid or challenge it when

- Visual generation is mistakenly expected from the encoder alone

- Input resolution removes required details

- A generic image benchmark substitutes for the actual domain

## Illustrative published families

- • Vision Transformer (ViT)

- • Convolutional and hierarchical vision backbones

- • The image tower in CLIP

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures