state-space-recurrent-models
State-space & recurrent models
📂 model-architectures
MODEL ARCHITECTURES#
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
MENTAL MODEL#
Stream the sequence through a learned state transition; the state is a compressed memory of the past.
DATA FLOW#
-
Current token + prior state
-
Learned state update
-
Selective gate or recurrent mixer
-
Output hidden state
-
Carry state forward
How it trains#
Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.
How inference runs#
Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.
Strengths#
-
• Linear-time sequence processing in the core recurrence
-
• Compact streaming state at inference
-
• Natural fit for long or continuously arriving sequences
Trade-offs#
-
• Compressed state can lose precise distant details
-
• Tooling and optimized kernels may be less mature than standard attention
-
• Architecture-level complexity claims must be verified end to end
Use it when#
-
Streaming or long-sequence efficiency is a first-order requirement
-
The target runtime has optimized kernels
-
Recall, quality, and throughput are benchmarked on the real workload
Avoid or challenge it when#
-
Exact arbitrary retrieval from long context is assumed without testing
-
The serving stack cannot exploit the architecture
-
A mature Transformer deployment already meets the budget
Illustrative published families#
-
• Mamba selective state-space models
-
• RWKV-style recurrent language models
-
• Hybrid attention–SSM stacks