Code Benchmarks

For Kode, these benchmarks are the most relevant — they measure real coding capability.

---

HumanEval

• arXiv: 2107.03374 (Chen et al., OpenAI, 2021)
• Size: 164 Python problems
• Format: Docstring → implementation; verified by unit tests
• Metric: pass@k — probability that at least 1 of k samples passes
• Status: Saturated — top models reach 95%+
• Limitation: Too small; simple problems; Python only

HumanEval+

• Origin: EvalPlus (2023)
• Improvement: Additional (more rigorous) tests for the same problems
• Result: Reduces scores by ~5–15% — models that "pass" the original tests fail the new ones

---

MBPP — Mostly Basic Python Problems

• arXiv: 2108.07732 (Austin et al., Google, 2021)
• Size: 974 Python problems (378 in the test set)
• Difficulty: Basic to intermediate
• Status: Saturated (GPT-4: 90%+)

MBPP+

• Additional tests by EvalPlus; more rigorous

---

SWE-bench — Software Engineering Benchmark

SWE-bench Original

• arXiv: 2310.06770 (Princeton, 2023)
• Size: 2,294 real GitHub issues (12 Python repositories)
• Format: Issue text → patch that resolves the issue + regression tests
• Evaluation: The patch generated by the model is applied and the existing tests run

SWE-bench Verified

• Version: 500 issues manually validated by humans (removed ambiguous ones)
• Why use it: More reliable scores; less noise
• Current reference: The standard benchmark for coding agents

SWE-bench Lite

• 300 "easier" problems; used for fast iteration

SWE-bench Multimodal

• Includes issues with screenshots and diagrams

---

LiveCodeBench

• URL: livecodebench.github.io
• Mechanism: Continuously collect new problems from LeetCode, Codeforces, AtCoder after the models' cutoff date
• Anti-contamination: Problems are always newer than the models' training
• Format: Competitive programming; verified by tests
• Update: Monthly

---

BigCodeBench

• arXiv: 2406.15877
• Size: 1,140 Python problems
• Differentiator: Uses real libraries (numpy, pandas, requests, PIL, etc.) — not just the stdlib
• Validates: The ability to use external APIs correctly
• More realistic than HumanEval/MBPP for real-world code

---

RepoBench

• arXiv: 2306.03091
• Focus: Repository-level code completion (cross-file context)
• Formats: Retrieval, completion, pipeline (combined)
• Why it matters: Coding in real projects requires context from multiple files

---

CrossCodeEval

• arXiv: 2310.11248
• Languages: Python, TypeScript, Java, C#
• Focus: Code completion with cross-file dependencies
• Realism: Real GitHub projects

---

DS-1000

• arXiv: 2211.11501
• Focus: Data science — numpy, pandas, tensorflow, PyTorch, matplotlib, sklearn, scipy
• Size: 1,000 problems
• Realism: Extracted from real Stack Overflow

---

CRUXEval

• arXiv: 2401.03065
• Focus: Reasoning about code execution
  • Input prediction: Given the output, guess the input
  • Output prediction: Given the input, guess the output
• Skill tested: Semantic understanding of code (not just generation)

---

EvoEval

• Origin: 2024
• Mechanism: Evolves HumanEval across multiple dimensions: harder, creative, tool, time-space
• Why use it: Less contamination than the original HumanEval

---

CanItEdit / EditEval

• Focus: Code editing — given existing code + instruction, apply the change
• Relevant: For AI coding assistants that edit the user's code

---

Aider Polyglot Benchmark

• URL: aider.chatdocsleaderboards
• Focus: Code editing across multiple languages (Python, JS, Go, Rust, etc.)
• Methodology: Models edit real code in a file; regression test
• Relevant: For coding assistants like Copilot/Kode

---

State-of-the-Art Table (April 2026)

---

Leaderboard Platforms