Overview
Vendors and labs open-sourcing evaluation harnesses that run coding agents on real product tasks (not only synthetic benches) — e.g. supabase-evals.
Timeline
- 2026-07-31: Supabase Evals Apache-2.0 + public leaderboard (2026-08-01-supabase-evals-open-source-benchmark)
Key Players
Analysis
Real-stack evals (MCP/CLI/containers) surface schema/RLS/docs-usage gaps that SWE-bench alone misses. Vendor-run leaderboards need attribution.