Skip to content

GitHub ships Copilot Evals in preview and benchmarks Copilot CLI against Claude Code

· by

The short version

GitHub's November 17 changelog put Copilot Evals into public preview for regression-testing custom agents and skills, and added a benchmarks-site view that compares Copilot CLI's resolution rate and cost per task against Claude Code and Codex CLI.

GitHub ships Copilot Evals in preview and benchmarks Copilot CLI against Claude Code

GitHub put a new testing tool in front of Copilot users on November 17: an in-editor way to catch regressions in a developer's own customized setup. The same day, GitHub also published the first head-to-head benchmark view lining up GitHub Copilot CLI's resolution rate and cost per task directly against Anthropic's Claude Code and OpenAI's Codex CLI running the same model.

What happened

GitHub Copilot Evals entered public preview covering the Individual tier plus the Business and Enterprise plans, per GitHub's changelog. Developers write a repeatable test, called an eval, for a custom skill, agent, or MCP tool wired into their Copilot setup, then rerun it after swapping a prompt or model to check whether the change helped or caused a regression. Approval is required for every run; nothing executes without a developer clicking go first. Evals runs on Vally, an extensible local evaluation engine, and installs through a VS Code extension, a Copilot CLI plugin, or a marketplace add-in inside the Copilot app.

GitHub's benchmarks site got its own addition that same day: a Harnesses view that puts Copilot CLI against Claude Code and Codex CLI at matched model and matched reasoning effort. Each harness result is plotted by resolution rate against mean cost per task, with lines connecting matched pairs and shaded ellipses showing the run-to-run variation observed across published runs. Paired differences then show Copilot CLI's resolution-rate edge, in percentage points, next to its cost savings, as a percentage, each with an approximate 95 percent bootstrap interval. GitHub frames the point of the exercise directly, writing that "much of the work we do to make Copilot better happens at the harness-level, and this view makes those improvements visible," rather than naming a single overall winner in the post itself; the per-model numbers sit inside the interactive chart on the benchmarks site. Only exact model-for-model pairings at medium reasoning effort make the cut. Copilot-only setups such as Copilot Fusion, and any result without a matching pair, are left out rather than stacked against a different underlying model. A second addition, Terminal-Bench-Science 0.1, scores agents on research workflows well outside ordinary software engineering, authored by domain experts and reviewed before publication, extending the site past pure coding benchmarks.

GitHub is now the one publishing the harness-versus-harness numbers

Buyers comparing coding agents have mostly had to piece together Copilot-versus-Claude-Code-versus-Codex comparisons from third-party leaderboards or a vendor's own favorable cherry pick. That changes who holds the data now. GitHub itself publishes head-to-head resolution-rate and cost figures against two named competitors, with confidence intervals attached, on its own benchmarks domain. A single highlighted run in a blog post is easy to cherry-pick; a dataset with bootstrap intervals and excluded mismatches is not, which gives a buyer evaluating harnesses on the same underlying model something closer to an audit trail than a marketing claim. Evals solves a narrower, separate problem for teams already committed to Copilot: catching a regression in a custom MCP tool or agent before it ships, instead of discovering it after a model or prompt swap already went live.

What to watch next

Anyone comparing Copilot CLI to Claude Code or Codex CLI at the same model should pull up the Harnesses view before trusting a vendor's own benchmark claim; it is the only place right now where a GitHub-published chart puts all three side by side with the same task set and the same reasoning effort.

Sources