General and Reasoning Benchmarks

Golden rule: Never include evaluation datasets in training — it contaminates the benchmark and invalidates comparisons.

---

MMLU — Massive Multitask Language Understanding

• arXiv: 2009.03300 (Hendrycks et al., UC Berkeley, 2020)
• Format: 15,908 multiple-choice questions (4 options)
• Domains: 57 subjects: STEM, social sciences, humanities, medicine, law, philosophy...
• Use: World-knowledge evaluation; main benchmark for general models
• Limitation: Saturated — top models exceed 90% (GPT-4o: 88%, Llama 3 70B: 82%)

MMLU-Pro

• arXiv: 2406.01574
• Improvement: 10 options (not 4); reasoning required; 12K questions
• Less saturated: Top models at ~65% (more discriminative)

---

ARC — AI2 Reasoning Challenge

• Source: Allen AI (2018)
• Format: 7,787 grade-school science questions (multiple choice)
• ARC-Easy: Answerable by simple retrieval models
• ARC-Challenge: Require reasoning; harder
• Status: Largely saturated (GPT-4: 96%)

---

HellaSwag

• arXiv: 1905.07830 (Zellers et al., 2019)
• Format: Complete an everyday activity (4 options)
• Difficulty: GPT-2 fails; GPT-4 gets >95% right
• Status: Saturated for modern models

---

WinoGrande

• arXiv: 1907.10641 (Sakaguchi et al., 2019)
• Format: Ambiguous pronoun resolution (Winograd schema)
• Size: 44,000 problems; adversarially filtered
• Status: Less saturated than HellaSwag; ~85% state of the art

---

TruthfulQA

• arXiv: 2109.07958 (Lin et al., OpenAI, 2021)
• Focus: Honesty — the model should answer truthfully, not reproduce popular myths
• Format: 817 questions; MC and free-generation evaluation
• Original finding: GPT-3 was right only 58% of the time (worse than humans)
• Status: Modern models reach 85%+ with RLHF

---

BIG-Bench / BIG-Bench Hard

• Repository: github.comgoogleBIG-bench
• BIG-Bench: 204 diverse tasks; collaboration of 444 researchers
• BIG-Bench Hard (BBH): 23 tasks where LLMs fall below humans
• Use: Still relevant for tasks that modern models have not saturated

---

GSM8K — Grade School Math

• Origin: OpenAI (2021)
• Size: 8,500 grade-school-level math problems
• Format: Natural-language answer; chain-of-thought is key
• Status: Top models get 95%+ right; saturated
• Replacement: MATH, AIME

---

MATH Dataset

• arXiv: 2103.03874 (Hendrycks et al., 2021)
• Size: 12,500 problems from math competitions
• Difficulty: 5 levels (1easy, 5olympiad)
• Areas: Algebra, Combinatorics, Geometry, Number Theory, Probability, Pre-calculus, Calculus
• Status: OpenAI's o3 model: 96.7%; still discriminative at levels 4–5

MATH-500

• Subset of 500 problems; frequently cited in papers

---

AMC / AIME — American Mathematics Competitions

AIME (American Invitational Mathematics Examination)

• Format: 15 questions; integer answer 0–999
• Difficulty: American olympiad
• AIME 2024: 30 problems (I + II)
• AIME 2025: 30 problems

---

Humanity's Last Exam (HLE)

• Origin: Scale AI + CAIS (2025)
• Size: 3,000 questions contributed by PhDs and experts
• Difficulty: Designed to be the "ceiling" — problems that expert humans take hours to solve
• Initial result: GPT-4o: 3.3%, Gemini 1.5 Pro: 2.5% — extremely difficult
• 2026 update: o3-high: ~18%, Claude Opus 4.7: ~14%
• Purpose: Replace saturated benchmarks; track frontier progress

---

GPQA — Graduate-Level Google-Proof Q&A

• arXiv: 2311.12022 (Rein et al., 2023)
• Size: 448 questions in biology, physics, chemistry
• Difficulty: Expert PhD students get ~65% right
• "Google-proof": Searching Google does not help
• Reference: o3: 87.7%, Claude Opus 4: 73.4%

---

DROP — Discrete Reasoning Over Paragraphs

• arXiv: 1903.00161
• Format: Reading and computation: extraction, mathematical operations, sets
• Use: Numerical reasoning over text

---

ARC-AGI — Abstraction and Reasoning Corpus for AGI

• Creator: François Chollet (creator of Keras — see 07-frameworks/distributed-training.md), 2019
• Paper: On the Measure of Intelligence (arXiv 1911.01547) — Chollet proposes measuring intelligence by skill-acquisition efficiency, not by performance on known tasks
• Format: Visual colored-grid patterns (up to 30×30) — input + output from a few examples; infer the rule and apply it to a new input
• Philosophy: Resist "scale brute-force" — tasks designed to require generalization over core knowledge priors (objectness, symmetry, counting, basic topology) that humans have innately; models that merely memorize massive patterns fail
• Difficulty: 8-year-old children get ~85% right on the semi-private set; initial GPT-4o (2024): ~2%; traditional LLM scaling hits a wall
• Evolution:
  • ARC-AGI-1 (2019): 1000 tasks (400 public training + 400 public evaluation + 200 private). Pre-2024 SOTA stagnant at ~30%.
  • ARC-AGI-2 (2024): Even harder. SOTA Dec/2024: o3-high (high-compute mode with extensive reasoning) ~76% on the semi-private; humans ~98%.
  • ARC-AGI-3 (announced 2025, in development): Chollet announces a next generation focused on interactive agentic tasks (not just static frame I/O). Even more resistant to brute-force.

ARC Prize (arcprize.org)

• URL: arcprize.org · Leaderboard: arcprize.org/leaderboard · GitHub: arcprize/ARC-AGI
• Organization: ARC Prize Foundation, founded by François Chollet + Mike Knoop (co-founder of Zapier) in 2024
• Competition structure (annual):
  • Grand Prize (US$ 600,000): first to reach ≥ 85% on the private evaluation set (not yet claimed in 2025-2026)
  • Top score, top paper, efficiency prizes: smaller awards (~US$ 50k each)
  • Total prize pool: US$ 1,000,000+ per edition
• Compute rules: evaluation run in a controlled environment with a compute limit (Kaggle notebook ~12h, no internet) — discourages "throw US$ 350k of o3-high inference" since it does not qualify for the Grand Prize
• ARC-AGI Pub track: Results from high-compute models (o3, Claude, Gemini) reported publicly without qualifying for the Grand Prize, in a separate leaderboard
• Why it matters:
  • Counter-narrative to "scaling solves everything": Chollet is vocal against the thesis that bigger LLMs → AGI; ARC-AGI is the operational form of that argument
  • Public open-source validation: all top-score solvers publish code (DSL search, program synthesis, neuro-symbolic hybrid)
  • History of winning approaches: showed that program synthesis + search + LLMs (not pure end-to-end LLM) is the path that advances the most
• Top approaches (2024-2025):
  • Jeremy Berman (top human interpretable solver, 2024) — handcrafted DSL + search
  • Greenblatt approach — GPT-4 + massive sampling + verification
  • MindsAI / Architects of Intuition — neuro-symbolic
  • o3 (OpenAI, Dec/2024) — first model to break 75% (in high-compute, cost ~US$ 350k for a complete evaluation)

For Kode: ARC-AGI is the canonical reference for evaluating real generalization vs memorization. If Kode wants a reasoning benchmark for internal use outside Code/Math, the ARC-AGI Pub leaderboard is a good baseline. Internally: study program synthesis + verification approaches — relevant for code generation with test-time verification (o-series paradigm).

---

Chatbot Arena (LMSYS / LMArena)

• URL: lmarena.ai (formerly: chat.lmsys.org/leaderboard)
• Methodology: Real users blindly compare two models → Elo rating
• Metric: Bradley-Terry Elo
• Why it is valuable: Reflects real human preferences; hard to "teach to the test"
• Limitation: Verbosity bias; English-centric

---

State-of-the-Art Table (April 2026)