mixture-of-experts-moe
Mixture of Experts (MoE)
📂 model-architectures
MODEL ARCHITECTURES#
Mixture of Experts (MoE)
Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.
MENTAL MODEL#
Large parameter capacity with sparse per-token activation; sparse compute is not the same as small memory or simple serving.
DATA FLOW#
-
Token hidden state
-
Router scores experts
-
Top-k expert dispatch
-
Expert transformations
-
Weighted merge + residual path
How it trains#
The main task loss trains experts and router together. Load-balancing or routing regularizers discourage collapse into a few experts; distributed implementations must coordinate token dispatch.
How inference runs#
Each token visits selected experts. Arithmetic can remain sparse while model weights, inter-device communication, routing imbalance, and batching still shape latency and cost.
Strengths#
-
• More parameter capacity without activating every parameter per token
-
• Experts may specialize through learned routing
-
• Scales naturally inside Transformer feed-forward blocks
Trade-offs#
-
• Large weight memory and distributed communication
-
• Routing imbalance and capacity overflow complicate training
-
• Reported total parameters do not reveal active compute or serving efficiency
Use it when#
-
Large-scale training and serving infrastructure can exploit sparse routing
-
Capacity is more constrained than per-token arithmetic
-
You can benchmark the actual hardware topology
Avoid or challenge it when#
-
A single-device or memory-constrained deployment is required
-
“Sparse” is being assumed to guarantee lower latency
-
Operational simplicity is more valuable than extra capacity
Illustrative published families#
-
• Switch Transformer
-
• Sparse expert layers inside language or multimodal models