# State-space & recurrent models


📂 model-architectures

## MODEL ARCHITECTURES

# State-space & recurrent models

Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.

## MENTAL MODEL

Stream the sequence through a learned state transition; the state is a compressed memory of the past.

## DATA FLOW

- Current token + prior state

- Learned state update

- Selective gate or recurrent mixer

- Output hidden state

- Carry state forward

## How it trains

Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.

## How inference runs

Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.

## Strengths

- •
Linear-time sequence processing in the core recurrence

- •
Compact streaming state at inference

- •
Natural fit for long or continuously arriving sequences

## Trade-offs

- •
Compressed state can lose precise distant details

- •
Tooling and optimized kernels may be less mature than standard attention

- •
Architecture-level complexity claims must be verified end to end

## Use it when

- Streaming or long-sequence efficiency is a first-order requirement

- The target runtime has optimized kernels

- Recall, quality, and throughput are benchmarked on the real workload

## Avoid or challenge it when

- Exact arbitrary retrieval from long context is assumed without testing

- The serving stack cannot exploit the architecture

- A mature Transformer deployment already meets the budget

## Illustrative published families

- • Mamba selective state-space models

- • RWKV-style recurrent language models

- • Hybrid attention–SSM stacks

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures