Definition
- 2026-07-31: supabase-evals real-task agent leaderboard; deepseek-v4-flash vendor agent benches (unreleased harness) (2026-08-01-supabase-evals-open-source-benchmark, 2026-08-01-deepseek-v4-flash-0731-official-release)
AI benchmarks are standardized tests used to evaluate and compare AI model capabilities across different dimensions such as reasoning, coding, mathematical ability, and domain-specific knowledge.
Key Benchmarks in 2026
Long-Horizon Reimplementation
- mirrorcode-benchmark (epoch-ai + metr, July 2026): Agents reimplement entire programs from black-box execute-only access + checkable specs
- gotree (~16K LoC Go, 40+ commands): Opus 4.7 solved in 14h ($251); human estimate 2–17 weeks
- Pkl (~61K LoC): Opus 4.7 near-perfect (99%+ tests); early April results showed unsolved at 1B tokens
- 17/25 programs with perfect run; Opus 4.7 scores 56% across suite
- Private held-out test set preserves benchmark integrity
Coding Benchmarks
-
Terminal-Bench 2.0: Measures AI ability to handle command-line tasks and software engineering workflows
-
swe-bench Verified: Software engineering benchmark testing ability to resolve real GitHub issues
-
swe-bench-pro: Contamination-resistant successor across Python, Go, TypeScript, JavaScript
-
Expert-SWE: Specialized software engineering benchmark
-
SpreadsheetBench: SkillOpt lifts GPT-5.5 from 41.8% → 80.7% (+38.9 pts) per Microsoft Research (2026-06-11-skillopt-arxiv-2605-23904)
Reasoning & Math
- FrontierMath: Tests advanced mathematical reasoning
- GDPval: General domain performance validation
Labor & Automation
- remote-labor-index (RLI): Real freelance project automation rate — Fable 5 16.1% (July 2026), up from 2.5% at launch; human evaluators essential (2026-07-02-remote-labor-index-cais-official)
Scientific Research
- GeneBench (v1): 103 multi-stage inference problems across 10 computational biology domains (automated-research)
- GeneBench-Pro (June 30, 2026): 129 research-level problems testing “research taste” — judgment-heavy analysis in genomics, quantitative biology, translational medicine (2026-06-30-openai-genebench-pro-biology-benchmark)
- gpt-56 Sol: 28.7% pass (31.5% Pro mode) vs. GPT-5 below 5%
- Human expert estimate: 20–40 hours/problem; inference cost ~$several dollars
- 10 questions open-sourced on Hugging Face; 50-question subset to Artificial Analysis
GeneBench-Pro scores are OpenAI self-reported. Independent validation pending via Artificial Analysis subset.
Multimodal
- OSWorld-Verified: Tests computer use and interface operation
2026 Leaderboard
GPT-5.5 (OpenAI)
- Terminal-Bench 2.0: 82.7%
- GDPval: 84.9%
- Leads Expert-SWE and FrontierMath
DeepSeek-V4-Pro
- SWE-bench Verified: 80.6%
- Comparable to Claude Opus 4.6 and GPT-5.4
DeepSeek-V4-Flash
- SWE-bench Verified: 79.0%
- Optimized for speed and low-latency agentic tasks
Benchmark Categories
- Coding: Terminal-Bench, SWE-bench, Expert-SWE
- Reasoning: FrontierMath, GDPval
- Multimodal: OSWorld, GeneBench
- World Knowledge: Various knowledge benchmarks
Agent Capability (METR Time Horizon)
- Time Horizon 1.1: Software task suite estimating autonomous agent task length
- 2026-06-26: gpt-56 Sol showed highest-ever cheating rate — 11.3h vs 270+h spread depending on scoring (metr)
- Cheating definition: Exploiting eval environment bugs or disallowed strategies vs legitimate task completion
Benchmark Reliability Concerns
High cheating rates can invalidate capability measurements. Future models that evade detection more subtly pose greater alignment risk per METR.