Innovation: Projects Q, K, V into a compact latent space; reduces the KV cache via low-dimensional projection
Performance: 93% KV cache compression; 1.4× speedup for smaller models
Pioneer: DeepSeek-V2/V3 — uses MLA as an architectural differentiator
DeepSeek-V4 abandoned MLA in favor of Hybrid CSA+HCA (next entry).
NSA — Native Sparse Attention (DeepSeek, 2025)
arXiv: 2502.11089 · Venue: ACL 2025 (Long)
Authors: Yuan, Gao, Dai, Luo et al. (DeepSeek-AI + Peking University)
Premise: Post-training sparse attention (H2O, SnapKV, etc.) cuts the KV cache but rarely speeds up training. NSA is hardware-aligned + natively trainable end-to-end.
Dynamic hierarchical strategy with 3 parallel branches per token:
Coarse-grained token compression — compresses wide blocks to preserve global context
Fine-grained token selection — selects top-k individual tokens for local precision
Sliding window — preserves immediate local dependencies
Result: a model pre-trained with NSA matches or surpasses Full Attention on general, long-context and instruction-reasoning benchmarks. Substantial speedups at 64k tokens in decodingforwardbackward.
Importance for the compendium:Direct predecessor of the CSA (Compressed Sparse Attention) used in DeepSeek-V4 — V4 cites "DeepSeek-AI 2025" for the DSA strategy applied within CSA. NSA established the "compression + selection" pattern that CSA refined and HCA extended.
Hybrid CSA + HCA (DeepSeek-V4)
Paper: DeepSeek-V4 §2.3 (04242026)
Compressed Sparse Attention (CSA): compresses every m tokens into 1 KV entry via a learned softmax-gate with positional bias; then applies DeepSeek Sparse Attention (DSA) — a lightning indexer (low-rank queries in FP4 + ReLU) selects the top-k blocks per query.
Heavily Compressed Attention (HCA): the same compression scheme with factor m' ≫ m, but dense (no sparse selection); interleaved with CSA layers.
Complementary attention: sliding-window branch (n_win recent uncompressed KVs), learnable attention sink, partial RoPE (only on the last 64 dims).
Result at 1M tokens: ~27% of single-token FLOPs and ~10% of the KV cache vs DeepSeek-V3.2; ~2% of the BF16 GQA8 baseline — makes 1M context viable on current hardware.
Complementary details in 03-models/open-source.md (DeepSeek-V4 section) and in 05-inference/kv-cache.md (Heterogeneous KV Cache).
Model: MiniMax-M3 (06012026) — see 03-models/open-source.md (MiniMax section).
Design premise: sparse attention by KV block selection — for each query it dynamically selects a subset of KV blocks instead of attending to the entire sequence. Described by MiniMax as a "clean and easily extensible sparse attention architecture."
Difference vs. the DeepSeek line: unlike MLA (which compresses QKV into a latent space) and CSA/DSA (which compress blocks before selecting), MSA operates on uncompressed KV — it selects KV blocks at full precision, avoiding the precision loss of compression in long context. It is the "selection without compression" point of the design space, complementary to NSA→CSA's "compression + selection."
Result at 1M tokens: compute per token ≈ 1/20 of the previous generation (M2); prefill >9× faster (9.7×) and decoding >15× faster (15.6×) vs. M2 at the same context.
Compendium status: technical report and weights promised for release ~10 days after launch (not published as of the recording date). The figures above are from MiniMax's official note; confirm the head/block counts when the tech report comes out.
FlashAttention 123
Problem: Standard attention is bounded by memory bandwidth (O(n²) in memory)
Solution: An IO-aware algorithm; computes attention in blocks — avoids materializing the full matrix in HBM
FlashAttention 1 (Dao et al., 2022) — arXiv:2205.14135 — the base