Agent and Long-Context Benchmarks

Agent Benchmarks

GAIA — General AI Assistants

• arXiv: 2311.12983 (Meta, HuggingFace, 2023)
• Size: 466 questions across 3 difficulty levels
• Format: Real questions requiring web search, code, files, multistep reasoning
• Humans: Score ~92%
• Top agentic LLMs: ~65% (Level 1), ~40% (Level 3)
• Why it's hard: Requires a real tool chain, not just an LLM

---

τ-bench (Tau-bench)

• arXiv: 2406.12045 (Sierra AI, 2024)
• Domains: Customer service (retail, airline)
• Format: Agent interacts with a simulated user + real database → resolves a ticket
• Measures: Precise tool use, consistency over long conversations, error correction
• Relevant: Enterprise scenarios; not trivially answerable with RAG

---

WebArena

• arXiv: 2307.13854 (CMU, 2023)
• Format: Real navigation of simulated websites (e-commerce, forums, code, email)
• Evaluation: Was the task completed correctly?
• Top score (2025): ~60% (with vision models)

---

VisualWebArena

• arXiv: 2401.13649
• Extension: WebArena with visual elements (images, charts, captchas)

---

OSWorld

• arXiv: 2404.07972 (2024)
• Format: Real desktop tasks (Linux, Windows, macOS) with screenshots
• Examples: "Open LibreOffice, create spreadsheet X, save as..."
• Top score: ~25% (very hard)

---

AgentBench

• arXiv: 2308.03688
• Domains: OS, DB, KG, alfworld, webshop, mind2web, housetour, webarena
• Format: Unified evaluation of agents across 8 domains

---

ToolBench

• arXiv: 2307.16789
• Focus: Use of 16,464 real APIs (RapidAPI)
• Evaluation: Agent selects and calls APIs correctly
• More realistic: 200K instructions with real tools

---

SWE-agent

• arXiv: 2405.15793
• System: Agent interface + GPT-4 for SWE-bench
• Mechanism: ACI (Agent-Computer Interface) optimized for code editing
• Initial result: 12.5% → basis for more modern systems

---

AppWorld

• arXiv: 2407.18900
• Focus: Agents in simulated apps (music, email, calendar, banking)
• Interaction: REST-style app APIs
• Realism: Everyday scenarios with multiple dependencies

---

Long-Context Benchmarks

RULER — What's the Real Limit of Long Context LLMs?

• arXiv: 2404.06654 (NVIDIA, 2024)
• Size: 4K to 128K tokens
• Tasks:
  • Single/Multi-hop NIAH (Needle In A Haystack)
  • Variable tracking
  • Multi-document QA
• Result: Most models degrade significantly above 32K tokens

---

HELMET — How to Evaluate LLMs on Long-Context Tasks

• arXiv: 2410.02694
• Tasks: RAG, book summarization, article citation, ICL with many examples
• Lengths: Up to 128K tokens
• Differentiator: Realistic tasks; not just NIAH

---

NIAH — Needle In A Haystack

• Concept: Hide a "needle" (a sentence with information) inside a long "haystack" (irrelevant text)
• Test: Can the model retrieve the information?
• Size: Typically tested from 1K to 1M tokens
• Tool: github.comgkamradtLLMTest_NeedleInAHaystack
• Limitation: Artificial test — doesn't reflect real long-context use

Variants

• Multi-Needle: Multiple needles in the same haystack
• Distractor: Haystack with contradictory information

---

ZeroSCROLLS

• arXiv: 2305.14196
• Focus: Summarization, QA, and reasoning over very long documents
• Datasets: GovReport, SumScroll, QASPER, QuALITY, Musique, SQuALITY, etc.
• Lengths: Up to 200K tokens

---

LOONG

• arXiv: 2311.04939
• Focus: Long, coherent reasoning (100K+ tokens)
• Task: Novel QA — questions about entire books

---

InfiniteRAG (2025)

• Focus: RAG over 1M+ token contexts
• Relevant to Kode: Ingesting entire repositories as context

---

Memory Analysis in Agents

MemGPT

• arXiv: 2310.08560
• Idea: An operating system for LLMs — explicitly manages short- and long-term memory
• Mechanism: Context "paging"; hierarchical storage
• Relevance: Conceptual foundation for agents with persistent memory

---

SWE-Bench Pro

A harder version of the original SWE-bench; issues from more complex repositories, with lower risk of data contamination.

---

GDPval (OpenAI)

• Origin: OpenAI (2026), internal
• Focus: Professional knowledge-work tasks (analysis, technical writing, research, strategic planning)
• Judges: Human domain experts (not LLM-as-judge)
• GPT-5.4: 83% accuracy — a record at the time of launch

---

OSWorld-Verified / WebArena Verified

Audited variants of the original benchmarks, with tasks manually verified to ensure solvability and correctness of the evaluation.

• GPT-5.4: Record on both at launch (March 2026)
• Focus: Computer use — real desktop and browser automation

---

Agent-SafetyBench

• Focus: Safety evaluation for autonomous agents
• Scale: 349 interaction environments; 2,000 test cases; 8 risk categories
• Coverage: Largest safety evaluation for agents published through 2026

---

CUB — Computer-Use Benchmark

• Focus: Unified for computer use (desktop + browser + terminal)
• Growing adoption: Together with GAIA, it became an independent reference for agents in 2025

---

Table: Model Scores on Agentic Benchmarks (2026)

---

Agent Leaderboard Platforms