Multimodal and Scientific Benchmarks

Vision-Language Benchmarks

MMMU — Massive Multidiscipline Multimodal Understanding

• arXiv: 2311.16502
• Size: 11,500 questions with images across 30 disciplines
• Format: Multiple choice; requires understanding image + text together
• Domains: Arts, sciences, medicine, engineering, humanities
• SOTA (2026): GPT-5: 82%, Gemini 3 Pro: 79%

MMMU-Pro

• arXiv: 2409.02813
• Improvement: 10 options (not 4); more complex problems; OCR required
• Less saturated: Top models ~60%

---

MMBench

• arXiv: 2307.06281 (Shanghai AI Lab, 2023)
• Size: 3,000 questions across 20 visual skills
• Skills: Object attributes, spatial relation, comparative reasoning, etc.
• Versions: English, Chinese, Dev/Test

---

VQAv2

• Source: VQA v2.0 (2017)
• Format: Open-ended questions about COCO images
• Size: 1.1M questions, 265K images
• Status: Saturated — top models score 85%+
• Replacement: MMMU, MMStar

---

DocVQA

• arXiv: 2007.00398
• Focus: QA over documents (PDFs, scans, forms)
• Size: 50,000 questions across 12,767 documents
• SOTA: 94%+ (Gemini 2.5 Pro, GPT-5)

---

ChartQA

• arXiv: 2203.10244
• Focus: Reasoning over graphs and charts
• Size: 9,608 questions across 4,804 charts
• Requires: OCR + quantitative reasoning

---

OCRBench

• arXiv: 2305.07895
• Focus: OCR capability of VLM models
• Includes: Text, tables, mathematical formulas, historical documents

---

ScienceQA

• arXiv: 2209.09513
• Format: Multimodal QA in sciences (elementary/middle school)
• Multimodal: Images + text; explainability (CoT)

---

MMStar

• arXiv: 2403.20330
• Focus: Eliminate "language leakage" — questions that can be answered without seeing the image
• More rigorous: Requires genuine vision

---

Video Benchmarks

Video-MME

• arXiv: 2405.21075
• Size: 2,700 videos from 30s to 1h
• Subtasks: Visual perception, temporal reasoning, OCR in video
• SOTA: Gemini 2.5 Pro (native video); GPT-5

MVBench

• arXiv: 2311.17005
• Focus: 20 video understanding tasks

---

Scientific Benchmarks

MedQA (USMLE)

• arXiv: 2009.13081
• Format: Questions from the American medical exam (USMLE Step 1–3)
• Size: 12,723 questions in English
• Human threshold: ~60% to pass
• SOTA: GPT-5, Claude Opus 4.7: 90%+ (surpasses doctors)

PubMedQA

• arXiv: 1909.06146
• Format: Answer yesnomaybe based on PubMed abstracts
• Size: 1,000 questions annotated by experts

MedBench

• arXiv: 2023.xxxxx
• Language: Chinese; traditional + Western medicine
• Benchmark: For healthcare models in the Asian context

---

LegalBench

• arXiv: 2308.11462 (Stanford, 2023)
• Size: 162 legal tasks; 40,000+ examples
• Includes: IRAC reasoning, statutory interpretation, contract analysis
• Skills: Issue spotting, rule recall, analysis, conclusion

---

FinanceBench

• arXiv: 2311.11944
• Format: QA over real financial documents (10-K, 10-Q, earnings)
• Size: 150 high-precision questions

---

Leaderboard Platforms

LMSYS Chatbot Arena / LMArena

• URL: lmarena.ai
• Methodology: Blind votes from real users; Bradley-Terry Elo
• Dimensions: Multiturn, coding, math, vision, multilingual, hard prompts

Open LLM Leaderboard v2 (HuggingFace)

• URL: huggingface.cospacesopen-llm-leaderboard/openllmleaderboard
• Benchmarks: MMLU-Pro, BBH, GPQA, MUSR, MATH-lvl5, IFEval
• Focus: Open-source models

HELM (Stanford)

• URL: crfm.stanford.edu/helm
• Focus: Holistic evaluation — multiple metrics (accuracy, robustness, fairness, efficiency)

AlpacaEval 2.0

• Methodology: LLM-as-Judge with Claude Sonnet as judge
• Metric: Win rate vs GPT-4 Turbo
• Use: Evaluation of instruction/chat models

Arena-Hard

• Methodology: 500 hard prompts from Chatbot Arena; GPT-4o as judge
• Correlates well with real human preferences

---

State-of-the-Art Table for Multimodal Benchmarks (April 2026)