Skip to content
NewsResearch

Harvard study of 718 firms finds AI coding agents' output gains erased by code review

· by Pondero Newsdesk

The short version

A Harvard working paper using Jellyfish engineering-analytics data on 700,000+ workers at 718 firms finds AI coding agents raise code output 30 percent, but a 49 percent longer review cycle wipes out the gain at the firm level.

Harvard study of 718 firms finds AI coding agents' output gains erased by code review

AI coding agents make engineers write 30 percent more code, but a Harvard working paper finds that gain never shows up in how much software a firm actually ships, because the time to review a pull request rises 49 percent in response.

What the study measured

Harvard researchers Fiona Chen and James Stratton analyzed a dataset from engineering-analytics platform Jellyfish covering 300 million work events, including GitHub commits, pull requests, and Jira issues, across more than 700,000 workers at 718 software firms from January 2021 through March 2026, per the paper. Using a staggered difference-in-differences design that compares firms as they adopt AI tools at different times, they found AI coding assistants (autocomplete-style tools) raised lines of code written by 12 percent, with only the effect on commits reaching statistical significance. Fully agentic tools, which write and submit code with less human direction, had a much larger effect: lines of code up 30 percent, commits up 20 percent, pull requests up 23 percent, all statistically significant, per the paper. Neither tool category produced a measurable change in resolved Jira issues or epics, the study's proxies for shipped software, or in total employment at the firm or engineering-team level.

The paper traces the gap to the review stage. After agent adoption, the average time to review a pull request rose 49 percent, the share of pull requests sent back for changes nearly doubled, and comments left per pull request rose 35 percent, per the paper. Firms responded by shifting more staff into review work: the share of employees doing code review grew 14 percent. The researchers note the bottleneck held even at firms that had already adopted AI code-review tools themselves, since human reviewers kept the final sign-off. To corroborate the pattern, the authors interviewed 21 developers, product managers, and engineering leads. One senior engineer said a tool like Claude Code "can write a draft in two seconds that's correct 90 percent of the time" for routine work but "makes frequent mistakes" on harder tasks, "and you end up spending more time reviewing," per the paper's interview transcripts, reported by t3n.

Seat spend now has to clear a review-capacity bar, not just a coding one

For teams justifying Copilot, Cursor, or Claude Code budgets on raw code-output metrics, the study is evidence that those metrics overstate the payoff unless review capacity scales alongside them. An engineering org that buys more agent seats without adding reviewers, or without training agents to produce review-ready diffs, risks the exact pattern Chen and Stratton measured: more commits, same number of shipped features, and reviewers absorbing the slack. The paper frames code review as a persistent bottleneck task, the kind of constraint that limits how much a productivity gain in one part of a pipeline shows up in the finished product.

What to watch next

Watch whether Jellyfish, LinearB, or Code Climate publish their own breakdown of the review-bottleneck pattern using fresher data, since the Harvard dataset ends in March 2026 and newer agent releases could shift the ratio. Also watch whether vendors selling AI code-review tools cite this paper, given it found the bottleneck persisted even where review automation had already been adopted.

Sources