Part V · 5 — Benchmarks and leaderboards
"Without good evals, you navigate in the dark." How to measure capability for real — and why private benchmarks are your only trustworthy compass.
🎨 Figure
F-V.5— The evaluation dashboard. Brief: a stylized "dashboard" with gauges of several benchmarks and a red "contamination" alert over the public ones; alongside, a vault labeled "private benchmark". Compendium palette.
5.1 Reference benchmarks
| Benchmark | Focus | Note |
|---|---|---|
| HumanEval / HumanEval+ | Python functions | classic, saturated |
| MBPP / MBPP+ | basic Python | — |
| SWE-bench / Verified / Multimodal | real GitHub issues | the most important today |
| LiveCodeBench | recent contests | avoids contamination |
| BigCodeBench | realistic library calls | — |
| RepoBench / CrossCodeEval | multi-file | — |
| GPQA Diamond | PhD-level science | — |
| Humanity's Last Exam | PhD-level, multidisciplinary | the frontier already matches the average human |
Current leaderboards (snapshot) live in Part III, doc 08-state-of-the-art-2026.kmd (SWE-bench Verified, Arena Elo).
5.2 The contamination trap
Critical tip: the public benchmarks are partially or fully contaminated in the training of frontier models. A high score may be memorization, not capability.
Build your own private benchmarks with data from your target domain:
- they represent the real problem you want to solve;
- no public model was trained on them → they measure genuine capability;
- they are your evaluation advantage (and, ultimately, your product advantage).
5.3 How to evaluate well
- Public + private combined; never public-only.
- Red teaming and uplift studies (does the model really help?).
- For reasoning/code: verifiable reward (did it pass the tests?) is the
gold standard — and becomes an RLVR signal (see
01-frontier-levers.kmd).
Evaluation is not a final stage — it is a continuous compass. It connects to stage 5 (Evaluation) and 5.5 (Homologation) of the lifecycle (Part I).