Part IV · Ch. 13 — SSM / Mamba
Connectionist · Long sequences · Selective state space model. Alternative to the Transformer with linear cost in sequence length. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.13.0— State scan. Brief: a long sequence traversed by a state that updates linearly, contrasted with the Transformer's quadratic attention matrix (side by side).
1. Definition and short history
Family based on state spaces (control theory) adapted to deep learning: S4 (2021) → Mamba (2023, input-dependent selection). It contests the niche of very long context.
2. Foundations
- Control theory / dynamical systems — continuous state equations.
- Signal processing — convolution/recurrence; impulse response.
- Linear algebra — SSM discretization; parallel scan.
- Information theory — compressing history into a fixed-size state.
3. Algorithms and architectures
- SSM —
x' = Ax + Bu; y = Cx + Du, discretized. - S4 — structured parametrization (HiPPO) for the long range.
- Mamba — selection (parameters depend on the input) + hardware-efficient
parallel scan.
- Hybrids — interleave SSM and attention blocks.
4. Inputs
- Hardware: GPU; optimized scan kernels (memory efficient).
- Data: long sequences (text, DNA, audio).
- Data structures: state tensors; no growing KV-cache.
- Systems: implementations with custom selective scan.
5. Specialized lifecycle
| Stage | Specialization |
|---|---|
| 0 Problem | Very long context, linear cost, streaming |
| 1 Data | Long sequences; same practices as LLM |
| 2 EDA | Length distribution, long-range dependencies |
| 3 Modeling | Pure SSM vs hybrid with attention; state size |
| 4 Training | Like LLM; parallel scan during training |
| 5 Evaluation | Long-context tasks (needle-in-haystack), perplexity |
| 5.5 Homologation | Stability on very long sequences |
| 6 Production | Inference with fixed state (constant memory) — advantage |
| 7 Monitoring | Quality vs length |
| 8 Retraining | Like LLM |
| 9 Governance | Like LLM |
6. Capabilities, modes and modalities
Efficient sequential: long context (documents, genomics, audio) with constant memory at inference; modes similar to the LLM.
7. Limits, risks and ethics
Newer and less mature than the Transformer; the advantage depends on the task; the ecosystem is still forming. General risks inherited from language models.
8. State of the art and examples
Mamba/Mamba-2, Jamba (SSM-Transformer hybrid); an active area of architecture research aiming to overcome the quadratic cost of attention.