Skip to content
NewsResearch

Study finds coding-agent harness choice swings cost 3x without improving accuracy

· by Pondero Newsdesk

The short version

Researchers at Shandong University ran Claude Code, OpenCode, and mini-SWE-agent against five models on nearly 500 SWE-bench tasks and found cost per task varied up to threefold for the same model while success rates stayed within two percentage points.

Study finds coding-agent harness choice swings cost 3x without improving accuracy

Swapping the scaffolding around a coding model, not the model itself, can triple what a task costs to solve while leaving the odds of solving it almost unchanged, according to a new study from researchers at Shandong University, per arXiv.

What

The researchers, Yangze Liu and Zhongyi Han, held five models fixed and ran each through three production agent harnesses, Claude Code, OpenCode, and the minimalist mini-SWE-agent, across 492 offline SWE-bench Verified tasks, per the paper. On the larger 447-task pool, tested with two Qwen models, Claude Code and mini-SWE-agent landed within two percentage points of each other and were statistically equivalent within a five-point margin, per VentureBeat's report. Cost told a different story. On GLM-5.3-Flash, a trial in Claude Code cost nearly twice as much as the same trial in mini-SWE-agent and more than three times as much as in OpenCode, the researchers found. Much of that gap traced to the first API call: Claude Code's system prompt and tool schemas front-loaded 16,581 tokens, versus 7,025 for OpenCode and 829 for mini-SWE-agent, an overhead that compounds every time the agent resends the growing conversation transcript.

A separate study from UC Berkeley's Sky Computing Lab, the HarnessTax project, reached a similar conclusion testing seven models across Claude Code, OpenAI's Codex CLI, and a minimal harness called Pi: harness choice shifted pass rates by only a few points but multiplied inference costs up to fivefold, per VentureBeat's writeup of both papers.

Vendor benchmarks don't resolve at this scale

The Shandong team also found that swapping harnesses flips about 13% of the hardest tasks from pass to fail, the same rate seen when rerunning one unchanged harness twice, per VentureBeat. That means a 45-task leaderboard comparison cannot reliably tell a 13-point harness gap from ordinary run-to-run noise, and the researchers argue teams should rerun their own baseline twice before crediting any harness swap with a real win. For engineering leaders picking a coding-agent platform, the practical lesson is to budget by cost per task and to test lean tool schemas against the heavier default, rather than trusting a single-run benchmark score to pick a winner.

Cost rankings were not fixed across models, either. On DeepSeek-V4-Flash, mini-SWE-agent needed more model calls than Claude Code (141 versus 104) and ended up costing slightly more per trial, about 15.2 cents versus 14.6 cents, the researchers reported. The researchers also noted that 96% to 99% of input tokens in these loops hit the provider's prompt cache, so cheaper cached-token pricing scaled the bill without changing which harness came out most expensive for a given model.

What to watch next

Watch whether Anthropic, the OpenCode team, or the mini-SWE-agent maintainers publish a rebuttal or their own cost breakdown in response. Also worth tracking is whether the findings start showing up in enterprise RFPs for coding-agent platforms, where the VB Pulse survey the researchers cite found 21% of enterprises running agent platforms have no automated way to cap token spending.

Sources