# Mixture of Experts (MoE)


📂 model-architectures

## MODEL ARCHITECTURES

# Mixture of Experts (MoE)

Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.

## MENTAL MODEL

Large parameter capacity with sparse per-token activation; sparse compute is not the same as small memory or simple serving.

## DATA FLOW

- Token hidden state

- Router scores experts

- Top-k expert dispatch

- Expert transformations

- Weighted merge + residual path

## How it trains

The main task loss trains experts and router together. Load-balancing or routing regularizers discourage collapse into a few experts; distributed implementations must coordinate token dispatch.

## How inference runs

Each token visits selected experts. Arithmetic can remain sparse while model weights, inter-device communication, routing imbalance, and batching still shape latency and cost.

## Strengths

- •
More parameter capacity without activating every parameter per token

- •
Experts may specialize through learned routing

- •
Scales naturally inside Transformer feed-forward blocks

## Trade-offs

- •
Large weight memory and distributed communication

- •
Routing imbalance and capacity overflow complicate training

- •
Reported total parameters do not reveal active compute or serving efficiency

## Use it when

- Large-scale training and serving infrastructure can exploit sparse routing

- Capacity is more constrained than per-token arithmetic

- You can benchmark the actual hardware topology

## Avoid or challenge it when

- A single-device or memory-constrained deployment is required

- “Sparse” is being assumed to guarantee lower latency

- Operational simplicity is more valuable than extra capacity

## Illustrative published families

- • Switch Transformer

- • Sparse expert layers inside language or multimodal models

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures