KV Cache — Memory Management in Inference

What the KV Cache Is

Attention computes: Attention(Q, K, V). For each generated token, the K and V of all previous tokens must be in memory. This cache grows linearly with sequence length.

Formula: KV Cache Size = 2 × layers × heads × head_dim × seq_len × bytes_per_element

Example: Llama 3 70B, seq 8K, BF16 → ~2 × 80 × 8 × 128 × 8192 × 2 bytes ≈ 42 GB

Problem: For servers with multiple concurrent users, the KV cache competes directly with the model weights for GPU memory.

---

PagedAttention — Paged Management

• arXiv: 2309.06180 | SOSP 2023 (vLLM)
• Authors: Kwon et al. (Berkeley)
• Idea: Inspired by OS virtual memory paging
• Mechanism: KV cache stored in non-contiguous blocks (pages); virtual → physical mapping table
• Benefits:
  • Zero internal fragmentation: a block is allocated only when needed
  • Prefix sharing: common prefixes (system prompt) mapped to the same physical pages
  • Copy-on-write for beam search
• Impact: Foundation of vLLM; 24× more throughput than naive HuggingFace transformers

---

RadixAttention (SGLang)

• arXiv: 2312.07104
• Mechanism: Radix tree of KV cache blocks; automatic reuse of common prefixes across requests
• Use case: Same system prompt for multiple users → shared cache
• Result: 5.6× the throughput of PagedAttention for workloads with repeated prefixes
• Adoption: SGLang; inspired prefix caching in vLLM

---

Chunked Prefill (vLLM, SGLang)

• Mechanism: Splits the prefill (prompt processing) into smaller chunks to interleave with decoding of other requests
• Benefit: Reduces TTFT (time to first token) latency under high load; better GPU utilization
• Implementation: --enable-chunked-prefill in vLLM

---

Sparse KV Cache — Compression via Selective Attention

H2O — Heavy Hitter Oracle

• arXiv: 2306.14048
• Mechanism: Keeps only the K "heaviest" tokens (highest accumulated attention) + recent tokens
• Budget: Sets a per-layer token budget (e.g., 20% of the context)
• Result: 90% memory reduction with <5% quality drop

SnapKV

• arXiv: 2404.14469
• Mechanism: Identifies important positions via the attention pattern of the prompt's last block → retains those positions in the KV cache
• Result: Better than H2O; less loss on recall tasks

ScissorHands

• arXiv: 2305.17118
• Mechanism: Attention pivots persist throughout the sequence; discards non-pivot tokens
• Result: Similar to H2O with more stable selection

PyramidKV / PyramidInfer

• arXiv: 2406.02069
• Mechanism: Lower layers keep more tokens; higher layers keep fewer (pyramid)
• Result: Better preservation of global information

RazorAttention

• arXiv: 2407.15891
• Mechanism: Keeps KV only for "retrieval heads" (attention heads that perform information retrieval); discards from the rest
• Result: 70% less KV cache with minimal loss

---

Streaming LLM — Sliding Window with Anchors

• arXiv: 2309.17453
• Authors: Xiao et al. (MIT)
• Problem: Sliding-window attention loses the initial tokens → catastrophic quality loss
• Solution: Keeps "attention sinks" (first 4 tokens) + a recent sliding window
• Result: Continuous generation over sequences of any length without re-computation
• Adoption: transformers, llama.cpp

---

KV Cache Quantization

TurboQuant (Google, 2026)

• arXiv: 2504.19874
• ~3.5 effective bits for the KV cache; 6× less memory than FP16; zero quality loss
• See 05-inference/quantization.md for details

FP8 KV Cache

• Native support in vLLM, SGLang, TensorRT-LLM
• ~2× less memory than FP16

---

Multi-Head Latent Attention (MLA) — DeepSeek

• arXiv: 2405.04434 (DeepSeek-V2)
• Mechanism: Projects K,V to a much smaller-dimensional latent space before the KV cache
• Result: KV cache 93% smaller than equivalent Multi-Head Attention
• DeepSeek-V3: MLA with 576 latent dims vs 7168 in MHA → drastic reduction
• Limitation: Requires a change to the model architecture (not a post-hoc optimization)
• DeepSeek-V4 abandoned MLA in favor of hybrid CSA+HCA (next section).

---

Heterogeneous KV Cache (DeepSeek-V4) — Hybrid CSA + HCA + SWA

• Paper: DeepSeek-V4 §3.6 (2026-04-24)
• Code: huggingface.codeepseek-aiDeepSeek-V4-Protreemain/inference (MIT)
• Problem: V4's hybrid attention (CSA with factor m + HCA with factor m' ≫ m + SWA with n_win recent) produces KV entries with distinct sizes and policies per layer, violating PagedAttention's assumptions.
• Solution — two separate structures:
  1. Classical block cache — stores CSA Indexer KV + CSA Main KV + HCA KV; block size = lcm(m, m'), producing k1 = lcm/m CSA entries and k2 = lcm/m' HCA entries per block. Serves the SparseAttention kernel co-design (alignment with cache lines).
  2. State cache per-request, fixed — stores SWA KV (n_win recent) + uncompressed tail tokens not yet ready for compression. Treated as a state-space model: KV depends only on the current position.
• Mixed-precision storage: RoPE dims in BF16, the rest in FP8 → ~50% savings vs pure BF16.
• On-disk KV cache: compressed entries (CSA/HCA) and SWA have separate disk-persistence strategies for shared-prefix reuse — eliminates re-prefill on shared long prompts.
• Combined result (with CSA+HCA+mixed precision): KV cache at 1M tokens drops to ~2% of the equivalent BF16 GQA8 baseline; ~10× smaller than DeepSeek-V3 at 1M.
• Important: The concept is not "KV cache in 3 tiers by access frequency" (an imprecise description circulating in popular-science videos). It is a heterogeneous 2-component cache (block + state), with on-disk as a separate persistence layer.

---

Optimized Prefill Techniques

Cross-Attention KV Sharing

• Shared prefixes (system prompts) stay in the cache once and serve everyone

Prefix Caching (vLLM v0.5+)

• Prefix hash → reuses previously computed KV cache
• vLLM's implementation of RadixAttention

Automatic Prefix Caching (APC)

• vLLM detects prefixes automatically without explicit configuration

---

CPU Offloading of the KV Cache

• Problem: GPU memory exhausted for very long contexts
• Solution: Offload cold blocks to CPU DRAM or NVMe via PCIe
• Tools: Inferflow, FlexGen
• Trade-off: PCIe latency vs VRAM (PCIe 5.0: ~64 GBs vs HBM3e: 3.4 TBs — 50× difference)

---

Tiered KV Cache — A Family of Multi-Tier Systems

A family of works that extends the offloading concept into a formal hierarchy HBM → DRAM → SSD/Disk, with adaptive management across tiers. It is not a single technique, it is an architectural pattern.

TTKV — Temporal-Tiered KV Cache

• arXiv: 2604.19769 (2026)
• Mechanism: Organizes KV into temporal tiers with heterogeneous capacity, precision, and latency, aligned to the HBM–DRAM hierarchy. Co-designs tier layout, tier content, and tier interaction.
• Differentiator: integrates KV reduction (sparsity) with awareness of the memory hierarchy — does not treat the two separately.

IMPRESS — Importance-Informed Multi-Tier Prefix KV Storage

• Venue: USENIX FAST 2025
• Mechanism: 3 explicit tiers — GPU memory + CPU memory + disk. Selectively loads only the "important" KVs for prefill/decoding; attacks the disk bottleneck directly.

Kareto — Adaptive Multi-Objective Tiered Storage

• arXiv: 2603.08739
• Focus: Dynamic configuration of heterogeneous storage balancing cost, throughput, and latency under variable workloads. Explicit multi-objective optimization.

MTDS — Multi-Tier Dynamic Storage

• Venue: Complex & Intelligent Systems (Springer, 2025)
• Mechanism: Offloads KV from GPU VRAM to a local hierarchy; reduces both memory and computation on the GPU.

LMCache (product)

• Available on: AWS SageMaker HyperPod, Google GKE
• Tiers: HBM (Tier 1) → CPU RAM (Tier 2) → Local SSD (Tier 3)
• Use case: prefix cache shared across multiple inference workers; cross-request and cross-pod reuse.

Common pattern of this family: latency grows ~10–100× with each tier below (HBM 3 TBs → DRAM 100 GBs → SSD 5 GBs), but capacity grows 10–1000×. The promotioneviction policy is the key — typically based on (a) frequency (classic LRU/LFU), (b) attention importance (IMPRESS), or (c) temporal age (TTKV).

Distinct from DeepSeek-V4's "heterogeneous KV cache" (previous section): tiered KV is a storage hierarchy (same structure, distinct locations); heterogeneous KV is a variety of structures (CSAHCASWA with distinct schemas). Orthogonal — V4 could run on top of LMCache.

---

Summary Table

---

Recommended Stack for Kode

1. PagedAttention via vLLM or SGLang (base)
2. Prefix caching for shared system prompts (repository context)
3. FP8 KV cache (when hardware supports it)
4. TurboQuant for maximum compression in long repository contexts
5. Chunked prefill for consistent latency on a shared server