Skip to content
NewsProduct launch

Supabase open-sources coding-agent benchmark: Claude Code consults docs in under 40% of real task runs

· by Pondero Newsdesk

The short version

Supabase released an Apache-2.0 eval framework on July 31 that runs Claude Code, Codex, and OpenCode against real database tasks, publishing the first public leaderboard for coding-agent performance on Supabase infrastructure.

Supabase open-sources coding-agent benchmark: Claude Code consults docs in under 40% of real task runs

Most coding-agent benchmarks grade text on toy problems. Supabase's new framework spins up a real database environment, invokes actual MCP servers and CLI tools, and grades three agents on tasks drawn from real support tickets and bug reports. The result is the first public leaderboard for how Claude Code, Codex, and OpenCode perform on Supabase infrastructure.

What Supabase built

Supabase released supabase/evals under an Apache-2.0 license on July 31, per the official announcement. The framework launches a hosted-like Supabase stack and a local CLI project in containers, then routes each agent through the actual Supabase MCP server and CLI. Task scenarios cover building schemas, debugging failed Edge Functions, and fixing broken Row Level Security policies.

Scoring mixes deterministic checks with an LLM judge for outputs that require semantic interpretation. Agents get one retry before final grading. Two suites run in parallel: a benchmark suite published at supabase.com/evals covering breadth across product areas and builder stages, and a regression suite Supabase runs daily to monitor specific known failure modes without skewing public scores. The full source and task scenarios are on GitHub.

What the leaderboard data shows

The most concrete finding concerns documentation use. Codex-based agents read roughly 8 Supabase documentation pages per scenario; Claude Code reads roughly 2 per scenario. Claude Code consulted Supabase docs in fewer than 40% of test runs even when the Supabase skill was loaded, per Supabase's analysis. The more capable OpenAI model checked docs most consistently across all benchmarked agents.

Skills moved the needle at the margins rather than in bulk. In the Build stage, Opus 5 and Kimi K3 scored 100% with no skill loaded at all. Where skills made a difference: Sonnet 5 climbed from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%. Supabase concluded that agents are already broadly capable on Supabase tasks, and that skills mainly help in edge cases where models need to discard outdated pre-training knowledge.

Two task-specific failure patterns appeared across agents. Even in projects already using Supabase's declarative schema workflow, agents hand-wrote migrations instead of working from the single-source schema file. Supabase responded with updated skill guidance and ran evals to verify the correction held. On Edge Functions, agents reached for supabase-js authentication by hand rather than using the newer @supabase/server package, which prompted Supabase to publish a "which package to choose" guide for agents to reference.

Why it matters

For teams running Claude Code on Supabase today, the doc-check gap has a direct operational implication. An agent that skips documentation in 60% or more of runs is likely drawing on pre-training knowledge about Supabase APIs, which may be months out of date. Supabase is building a CLI command and an MCP tool so agents can file feedback when they get stuck, creating a feedback loop between agent behavior and documentation coverage.

The methodology itself is now a template other infrastructure vendors can follow: real containerized stack, real tool calls, tasks drawn from actual support tickets. If Neon, PlanetScale, or Convex publish comparable evals, teams will have cross-stack data for choosing which agent performs best against their specific infrastructure, something no existing benchmark currently provides.

What to watch next

Supabase said it plans to expand scenario coverage and graduate stable regression scenarios into the public benchmark. The live leaderboard at supabase.com/evals is the place to track whether the Claude Code documentation gap narrows in subsequent releases. Supabase's upcoming agent feedback tooling, once shipped, will also indicate whether the eval suite starts catching real-world failure modes before users encounter them.

Sources