Part V · 4 — Tools and frameworks
The practical stack of 2026 — what to use at each stage, from fine-tuning to observability. Principle: start simple; only adopt heavy frameworks when the pain justifies it.
🎨 Figure
F-V.4— The tool bench. Brief: a tools panel organized by stage (training, inference, sandbox, eval, observability), each one a labeled "instrument". Compendium palette.
4.1 Recommended stack (2026)
| Stage | Tool |
|---|---|
| Fine-tuning SFT | Axolotl (flexible) or LLaMA-Factory |
| DPO / GRPO | TRL |
| Fine-tuning on 1 GPU | Unsloth |
| Inference (production) | vLLM (high concurrency), SGLang (agents) |
| Single-user / local | llama.cpp / Ollama |
| Execution sandbox | E2B, Modal, or Docker with limits |
| Agent | your own ReAct loop (~200 lines); LangGraph only if you need it |
| Eval | SWE-bench runner, bigcode-evaluation-harness, lm-evaluation-harness |
| Data pipeline | datatrove (HF), datasketch, tree-sitter |
| Observability | LangFuse, Weights & Biases, Phoenix |
Golden rule: write your own 200-line ReAct loop before falling into heavy agentic frameworks. And never run LLM code outside a sandbox.
4.2 Inference efficiency (2025–2026 advances)
Inference is where the production cost lives. Three recent advances:
- TurboQuant (Google, ICLR 2026) — vector quantization of the KV cache to
3.5 bits/value: *6× less RAM* up to 8× faster on H100, without retraining or accuracy loss.
- EAGLE-3 (NeurIPS 2025) — speculative decoding with multi-layer fusion:
2–6× faster; integrated into vLLM, SGLang, TensorRT-LLM.
- P-EAGLE (2025) — generates all K draft tokens in a single forward;
1.69× faster than EAGLE-3 on the B200.
Complementary: TurboQuant reduces the memory of the KV cache; EAGLE-3 accelerates token generation. Base concepts (quantization, KV cache, speculative decoding) in the chapter LLM, §3.
4.3 Adoption philosophy
- Validate the product in the cloud with an API before training anything.
- Start with QLoRA of 7–32B before dreaming of 70B+.
- Invest in data and evals, not in exotic architecture.
- Treat safety (sandbox, red team) as part of the product from day 1.