vision-encoders
Vision encoders
📂 model-architectures
MODEL ARCHITECTURES#
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
MENTAL MODEL#
A visual reader. It creates representations; generation requires a decoder or generative model.
DATA FLOW#
-
Image or frames
-
Patch, convolutional, or hybrid stem
-
Vision backbone
-
Spatial tokens / feature pyramid / pooled vector
-
Task head or modality connector
How it trains#
Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.
How inference runs#
The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.
Strengths#
-
• Reusable visual features
-
• Efficient classification, retrieval, and perception
-
• Can connect images to language models through a projector or cross-attention
Trade-offs#
-
• Resolution and patch size determine lost detail and token cost
-
• Training objectives create different blind spots
-
• A pooled vector is insufficient for many spatial tasks
Use it when#
-
The system must understand or retrieve visual content
-
You can choose pooled versus spatial features based on the task
-
Small text, charts, and domain imagery are evaluated explicitly
Avoid or challenge it when#
-
Visual generation is mistakenly expected from the encoder alone
-
Input resolution removes required details
-
A generic image benchmark substitutes for the actual domain
Illustrative published families#
-
• Vision Transformer (ViT)
-
• Convolutional and hierarchical vision backbones
-
• The image tower in CLIP