Vibeleaderboard
← All Intel
Intel / article

Harvard researchers find AI coding agents shifted the bottleneck to code review

Source
complexdiscovery.com
Author
judahmeek
Date

Context

When companies give engineers AI coding agents, the engineers write much more code, but the firms ship only a little more finished software, and the difference piles up in code review. That is the central finding of a working paper by Harvard PhD candidates Fiona Chen and James Stratton, reported by ComplexDiscovery. The authors used data from Jellyfish, a platform companies use to track their engineering teams: 445 million work events, covering code in GitHub, tasks in Jira and meetings in Google Calendar, for 720,542 workers at 655 firms between January 2021 and August 2026.

The study compares firms that adopted AI tools earlier with firms that adopted later, relying on the authors' argument that buying software involves security reviews, legal approval and pilots whose length varies in ways unrelated to how fast a firm is growing. Twelve months after adopting AI coding assistants, which suggest edits as an engineer types, firms wrote 30.4% more lines of code and opened 20.9% more pull requests, the batches of changes submitted for review. Adopting agents, which can carry out larger tasks on their own, added 44.0% more lines and 35.7% more pull requests on top. Finished work moved far less. Jira issues resolved, which the authors count once the work reaches production, rose 13.7% after agents, a borderline significant result that rules out a rise above 30.7%, and changed by -1.5% after assistants. A machine-learning check found no sign that issues were getting larger, and the authors found no significant change in employment.

Review is where the gap shows up. After adoption, the time from submitting code for review to its approval rose 56.1%. Code waited 59.9% longer before review began, which the authors read as reviewers running out of capacity, and spent 58.5% longer in active review while drawing 38.6% more comments per change, which they read as a sign that the code arriving for review was of lower quality. AI review tools have not removed the constraint: 79.1% of firms had used them by July 2026, but only 23.3% of review comments were written by AI.

The conclusion that output-based measures overstate AI's gains is well supported for the firms studied; the authors say directly that lines of code and commits overstate them. Their limits matter for anyone generalizing. The estimates cover only the first 12 to 16 months after adoption, the sample is mostly medium-to-large established firms, the data come from a commercial partner that reviewed the manuscript only for disclosure of proprietary information, and the paper is a job-market paper rather than a peer-reviewed publication. Its long-run figures, such as output rising 57.8% by 2031 if review productivity stays fixed, come from a model simulation, not observation. ComplexDiscovery's article was written about an August draft whose figures differ from the November version cited here.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Read the source complexdiscovery.com
Recommended reads
Comments

Checking sign-in…

Loading comments…