Speculative Decoding and Inference Acceleration

The Problem: Autoregressive is Slow

LLMs generate 1 token per step. The bottleneck is memory latency (KV cache), not compute.

Solution: Generate multiple candidate tokens with a small model (draft) → verify all of them at once with the large model (verifier) → accept correct tokens in parallel.

Gain: 2–6× speedup without modifying the large model; identical quality.

Original Speculative Decoding

  • arXiv: 2211.17192 | ICML 2023
  • Authors: Leviathan et al. (Google)
  • Mechanism: Small draft model generates K tokens → target model verifies in 1 forward pass → accepts correct prefix
  • Condition: Draft and target must have the same output distribution (target decodes without bias)
  • Typical gain: 2–3× in latency

Speculative Sampling (DeepMind)

  • arXiv: 2302.01318
  • Alternative: Same idea, exact probabilistic sampling to preserve the distribution

Medusa — Multiple Prediction Heads

  • arXiv: 2401.10774
  • Mechanism: Adds N extra decoding heads to the model (no separate model)
  • Prediction: Each head predicts the token at position t+1, t+2, ..., t+N
  • Verification: Tree attention verifies multiple paths in parallel
  • Gain: 2–3× speedup; requires fine-tuning to add the heads
  • Advantage: No separate draft model; simple deploy

EAGLE — Speculative Decoding with Feature-Level Draft

EAGLE-1

  • arXiv: 2401.15077 | ICML 2024
  • Innovation: The draft model predicts features of the next token (embedding level), not tokens directly
  • Gain: 3–4× speedup vs autoregressive
  • Method: Auto-regression over features + LM head of the original model

EAGLE-2

  • arXiv: 2406.16858
  • Innovation: Adaptive draft tree — dynamically adjusts the number of candidates based on draft confidence
  • Gain: 3.5–5× speedup; better use of compute

EAGLE-3

  • arXiv: 2503.xxxxx | 2025
  • Innovation: Multi-layer feature aggregation — uses features from multiple layers of the target model
  • Gain: 4–6× speedup; state of the art (March 2025)
  • Improvement: Better draft quality on long reasoning tasks (CoT)

P-EAGLE (Parallel EAGLE)

  • arXiv: 2504.xxxxx | 2025
  • Innovation: Multiple instances of the draft model in parallel + pipeline
  • Gain: Up to 6× speedup on multi-GPU hardware
  • Trade-off: More memory (multiple draft instances)

Hydra — Multi-Head with Improved Training

  • arXiv: 2402.05109
  • Based on: Medusa
  • Improvement: Position-specialized heads; training with rejection sampling
  • Gain: Slightly superior to the original Medusa

Lookahead Decoding

  • arXiv: 2402.02057 (DeepMind 2024)
  • Mechanism: No draft model; uses Jacobi iteration to generate and verify tokens "in parallel" within the model itself
  • Advantage: Zero additional memory overhead
  • Gain: 1.5–2× speedup (less than EAGLE, but universal)

SpecInfer / REST

SpecInfer

  • arXiv: 2305.09781
  • Mechanism: Speculation tree with multiple draft models
  • Focus: Production serving system

REST — Retrieval-Based Speculative Decoding

  • arXiv: 2311.08252
  • Mechanism: Retrieves continuations from a corpus instead of using a draft model
  • Advantage: No draft model training cost; adaptable by domain

Multi-Token Prediction (MTP)

  • arXiv: 2404.19737 (Meta 2024)
  • Mechanism: Trains the model to predict N tokens at once with N output heads
  • Adoption: DeepSeek-V3 uses MTP as an auxiliary training objective
  • Benefit: Leverages it for speculative decoding at inference time without a separate draft model
  • Result: Improves representation quality + inference speed

QuantSpec (Apple, 2025)

  • arXiv: 2502.10424
  • Published: Apple Machine Learning Research
  • Mechanism: Self-speculative decoding with a 4-bit hierarchical KV cache and 4-bit weights for the draft model
  • Differentiator: Draft and target are the same model in different quantization configurations — no separate model
  • Speedup: ~2.5× with acceptance rate > 90%
  • Integration: Apple MLX framework

Comparison Table

Method Speedup Requires Fine-Tune? Extra Memory Production Support
Original Speculative Decoding 2–3× No Draft model vLLM, TGI
Medusa 2–3× Yes (heads) Minimal vLLM
EAGLE-2 3.5–5× Yes (draft model) Draft model SGLang, vLLM
EAGLE-3 4–6× Yes Draft model SGLang
Lookahead 1.5–2× No Zero llama.cpp
MTP (DeepSeek-V3) ~2× Training Zero (runtime) Native
QuantSpec ~2.5× No Zero MLX (Apple)

Integration in Inference Servers

  • vLLM: Native support for Medusa, EAGLE, SpecInfer, multi-step decoding
  • SGLang: EAGLE-2/3 natively integrated; best support for speculative decoding
  • TGI (HuggingFace): Speculative decoding with configurable draft model
  • llama.cpp: Lookahead decoding; draft model speculative

For Kode

  • Completion latency in IDE: EAGLE-2 with Qwen2.5-Coder-32B (target) + Qwen2.5-Coder-1.5B (draft) → target of <50ms per token
  • Server: SGLang with EAGLE-3 for maximum throughput
  • Local: llama.cpp with lookahead decoding (zero overhead)