Introducing Supabase Evals

date: 2026-07-31

Today we’re open sourcing supabase/evals, our benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real Supabase tasks, for example, building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, and then scores how well they performed. It powers both our published benchmark and an internal regression suite we monitor daily.

Why we built this

Agents are becoming a primary way people build with Supabase, interacting through our CLI, MCP server, agent skills, and docs. We needed a way to understand how agents perform across all of these Supabase surfaces, not just one tool in isolation.

How we approached it

Dimensions covered: product areas (Database, Auth, etc.), topics (SDK, observability), and stages (build, resolve issues). Scenarios split into:

• Benchmark scenarios — breadth; published on supabase.com/evals
• Regression scenarios — known failure modes; monitored daily; do not skew published scores

Every scenario runs against a real Supabase environment: hosted-like stack and local CLI project in containers so agents invoke the actual MCP server and CLI. Scoring combines deterministic checks and LLM-as-a-judge. Agents get one retry after failure before grading.

Key findings (snapshot at publication)

• Agents pass most scenarios with no skill loaded. In Build stage, Opus 5 and Kimi K3 both scored 100% unaided.
• Skills closed remaining gaps: Sonnet 5 78%→100%, GPT-5.6 Sol 89%→100%, GPT-5.4 mini 78%→89%.
• Weaknesses: agents hand-write migrations instead of declarative schemas; hand-verify auth instead of @supabase/server; docs usage uneven (Codex/GPT-5.6 ~8 pages/scenario vs Claude Code ~2; Claude Code checks docs in under 40% of scenarios even with skills).

License: Apache-2.0. Repo: https://github.com/supabase/evals