Transformer and Attention Mechanisms

The Original Paper

Attention Is All You Need (2017)

  • Authors: Ashish Vaswani et al. (Google Brain)
  • arXiv: 1706.03762
  • Contribution: Pure Transformer architecture — eliminates RNNs/CNNs; multi-head self-attention; enables full parallelism in training
  • Result: 28.4 BLEU on WMT14 En-De; 3.5-day training on 8 P100s

Attention Variants

Multi-Query Attention (MQA)

  • Paper: Noam Shazeer (2019)
  • Problem: KV cache grows with the number of heads — a memory bottleneck
  • Solution: A single shared K and V head; multiple Q heads
  • Speedup: Significant reduction of the KV cache; slight quality trade-off

Grouped-Query Attention (GQA)

  • arXiv: 2305.13245
  • Solution: An intermediate number of K/V groups (between 1 and full); recovers MHA quality at near-MQA speed
  • Speedup: 10.6× on LLaMA-2-7B at 8K context with minimal quality loss
  • Adopted by: Llama 3, Qwen 2+, Mistral, DeepSeek-V3

Multi-Head Latent Attention (MLA)

  • arXiv: 2502.07864 (TransMLA), 2502.14837
  • Innovation: Projects Q, K, V into a compact latent space; reduces the KV cache via low-dimensional projection
  • Performance: 93% KV cache compression; 1.4× speedup for smaller models
  • Pioneer: DeepSeek-V2/V3 — uses MLA as an architectural differentiator
  • DeepSeek-V4 abandoned MLA in favor of Hybrid CSA+HCA (next entry).

NSA — Native Sparse Attention (DeepSeek, 2025)

  • arXiv: 2502.11089 · Venue: ACL 2025 (Long)
  • Authors: Yuan, Gao, Dai, Luo et al. (DeepSeek-AI + Peking University)
  • Premise: Post-training sparse attention (H2O, SnapKV, etc.) cuts the KV cache but rarely speeds up training. NSA is hardware-aligned + natively trainable end-to-end.
  • Dynamic hierarchical strategy with 3 parallel branches per token:
    1. Coarse-grained token compression — compresses wide blocks to preserve global context
    2. Fine-grained token selection — selects top-k individual tokens for local precision
    3. Sliding window — preserves immediate local dependencies
  • Result: a model pre-trained with NSA matches or surpasses Full Attention on general, long-context and instruction-reasoning benchmarks. Substantial speedups at 64k tokens in decodingforwardbackward.
  • Importance for the compendium: Direct predecessor of the CSA (Compressed Sparse Attention) used in DeepSeek-V4 — V4 cites "DeepSeek-AI 2025" for the DSA strategy applied within CSA. NSA established the "compression + selection" pattern that CSA refined and HCA extended.

Hybrid CSA + HCA (DeepSeek-V4)

  • Paper: DeepSeek-V4 §2.3 (04242026)
  • Compressed Sparse Attention (CSA): compresses every m tokens into 1 KV entry via a learned softmax-gate with positional bias; then applies DeepSeek Sparse Attention (DSA) — a lightning indexer (low-rank queries in FP4 + ReLU) selects the top-k blocks per query.
  • Heavily Compressed Attention (HCA): the same compression scheme with factor m' ≫ m, but dense (no sparse selection); interleaved with CSA layers.
  • Complementary attention: sliding-window branch (n_win recent uncompressed KVs), learnable attention sink, partial RoPE (only on the last 64 dims).
  • Result at 1M tokens: ~27% of single-token FLOPs and ~10% of the KV cache vs DeepSeek-V3.2; ~2% of the BF16 GQA8 baseline — makes 1M context viable on current hardware.
  • Complementary details in 03-models/open-source.md (DeepSeek-V4 section) and in 05-inference/kv-cache.md (Heterogeneous KV Cache).

MSA — MiniMax Sparse Attention (MiniMax-M3, Jun/2026)

  • Model: MiniMax-M3 (06012026) — see 03-models/open-source.md (MiniMax section).
  • Design premise: sparse attention by KV block selection — for each query it dynamically selects a subset of KV blocks instead of attending to the entire sequence. Described by MiniMax as a "clean and easily extensible sparse attention architecture."
  • Difference vs. the DeepSeek line: unlike MLA (which compresses QKV into a latent space) and CSA/DSA (which compress blocks before selecting), MSA operates on uncompressed KV — it selects KV blocks at full precision, avoiding the precision loss of compression in long context. It is the "selection without compression" point of the design space, complementary to NSA→CSA's "compression + selection."
  • Result at 1M tokens: compute per token ≈ 1/20 of the previous generation (M2); prefill >9× faster (9.7×) and decoding >15× faster (15.6×) vs. M2 at the same context.
  • Compendium status: technical report and weights promised for release ~10 days after launch (not published as of the recording date). The figures above are from MiniMax's official note; confirm the head/block counts when the tech report comes out.

FlashAttention 123

  • Problem: Standard attention is bounded by memory bandwidth (O(n²) in memory)
  • Solution: An IO-aware algorithm; computes attention in blocks — avoids materializing the full matrix in HBM
  • FlashAttention 1 (Dao et al., 2022) — arXiv:2205.14135 — the base
  • FlashAttention 2 (2023) — better parallelism, better GPU occupancy
  • FlashAttention 3 (2024) — optimized for Hopper (H100); asynchronous pipelining; 75% of theoretical FLOPS
  • Integrated into: PyTorch, vLLM, SGLang, TensorRT-LLM, all major frameworks

Positional Encoding

RoPE — Rotary Positional Embedding

  • arXiv: 2104.09864
  • Mechanism: Encodes relative position via rotation matrices in complex space
  • Advantage: Naturally relative; smooth decay of attention with distance
  • Adopted by: Llama (123/4), Qwen, Mistral, DeepSeek, GPT-NeoX, PaLM

ALiBi — Attention with Linear Biases

  • arXiv: 2108.12409
  • Mechanism: Linear bias penalty on attention based on relative distance; no position embedding
  • Advantage: Extrapolation to contexts larger than the one trained

Positional Interpolation (PI)

  • Mechanism: Compresses position indices within the training window (scaling)
  • Extension: Allows extrapolating context 2-4× without retraining

YaRN (Yet Another RoPE Extension)

  • arXiv: 2309.00071 | ICLR 2024
  • Mechanism: Piecewise frequency scaling + temperature; NTK-by-parts interpolation
  • Efficiency: 10× fewer tokens, 2.5× fewer steps vs prior methods
  • Extension: 128K+ tokens on LLaMA-2 with 400-600 fine-tuning steps

LongRoPE

  • arXiv: 2402.13753 | ICML 2024
  • Extension: 2M tokens with only 1K fine-tuning steps
  • Integrated into: Microsoft Phi-3

Mixture-of-Experts (MoE) Architectures

Sparsely-Gated MoE (2017)

  • arXiv: 1701.06538
  • Authors: Shazeer et al. (Google)
  • Mechanism: A gating network selects the top-k experts per token; only k active per forward pass
  • Impact: Increases capacity 1000× with minimal compute overhead

Switch Transformer (2021)

  • arXiv: 2101.03961
  • Simplification: top-1 routing (only 1 expert) — more stable, less communication
  • Trained on: T5 11B → 1T parameters with the same FLOP

Mixtral 8×7B8×22B (Mistral, 20232024)

  • Architecture: MoE with 8 experts, 2 active per token (~12.9B effective compute per token)
  • License: Apache 2.0

DeepSeek MoE Architecture

  • Innovation: Fine-grained experts + shared experts; allows greater specialization with more efficient routing
  • Used in: DeepSeek-V2, V3, R1 (671B total, 37B active)

Tokenization

BPE (Byte Pair Encoding)

  • Origin: Philip Gage (1994) for compression; adapted by Sennrich et al. (2015) for NLP
  • Mechanism: Iteratively merges the most frequent byte pairs
  • Used by: GPT-234, Llama, Qwen, DeepSeek, most modern LLMs

SentencePiece

  • Paper: Kudo & Richardson (2018)
  • Differentiator: Language-agnostic; treats input as raw bytes without language pre-processing
  • GitHub: github.comgooglesentencepiece

WordPiece

  • Origin: BERT (Devlin et al., 2018)
  • Difference from BPE: Maximizes the corpus probability when selecting the vocab instead of frequency

Tiktoken

  • Origin: OpenAI
  • Used by: GPT-3.544o — a 100K-token vocabulary
  • Note: Efficient encoding for code (reduces tokens for whitespace/indentation)