Part IV · Ch. 12 — RNN / LSTM / GRU
Connectionist · Sequences · Recurrent network. Processes sequences while maintaining an internal state over time. Largely succeeded by the Transformer, but historical and still useful in niches. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.12.0— The temporal loop. Brief: a recurrent cell "unrolled" in time, with the hidden state passing step to step; LSTM gates (inputforgetoutput).
1. Definition and short history
A network with feedback that carries state. LSTM (1997) solved the vanishing gradient; it dominated NLP and time series until 2017 (see Part III, era 4).
2. Foundations
- Stochastic processes / time series — temporal dependence.
- Calculus — backprop through time (BPTT); vanishing/exploding gradient.
- Dynamical systems theory — recurrent state.
3. Algorithms and architectures
- Simple RNN — recurrent hidden state (struggles with the long term).
- LSTM/GRU — gates control long-term memory.
- Bidirectional / stacked — context from both sides, depth.
- Seq2seq + attention — the historical bridge to the Transformer.
4. Inputs
- Hardware: CPU/GPU; sequential training (less parallelizable than the Transformer).
- Data: sequences (text, sensors, finance).
- Data structures: sequential tensors; hidden state.
- Systems: PyTorch/TF; lightweight on edge (small models).
5. Specialized life cycle
| Stage | Specialization |
|---|---|
| 0 Problem | Short/streaming sequence, limited resources, time series |
| 1 Data | Aligned sequences; windows; temporal normalization |
| 2 EDA | Seasonality, trend, autocorrelation |
| 3 Modeling | LSTM/GRU, depth, bidirectional |
| 4 Training | BPTT; gradient clipping; teacher forcing |
| 5 Evaluation | Prediction/sequence error; horizon |
| 5.5 Acceptance | Streaming latency, stability |
| 6 Production | Efficient streaming inference; edge |
| 7 Monitoring | Temporal drift |
| 8 Retraining | Sliding window, new data |
| 9 Governance | According to domain (finance, health) |
6. Capabilities, modes and modalities
Sequential/temporal: time series, lightweight streaming, embedded control; still competitive where simplicity and latency rule.
7. Limits, risks and ethics
Limited long-term memory; sequential training poorly parallelizable; surpassed by the Transformer and by SSMs (ch. 13) on most tasks.
8. State of the art and examples
LSTM/GRU in time series and on devices; conceptual resurgence via SSMs (Mamba) that recover recurrent efficiency with a better long term.