Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests
Source
arxiv.org
Author
Wannita Takerngsaksiri, Nhat Duong, Scott Barnett
Date
Context
An accepted AI-generated code change can still create follow-up repair work. A study of merged pull requests followed 6,774 AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →-authored changes and 5,044 human-authored changes from the same open-source repositories and period. The authors report that agent changes had 1.62 times the odds of receiving a verified subsequent fix.
Much of that repair work was also attributed to agents: 69.6% of verified fixes following agent merges came from the same coding-agent system. The finding makes merge counts an incomplete measure of completed work. In this sample, agents frequently repaired their own changes, while those changes attracted fixes more often than the human comparison group.
Key takeaways · AI-distilled
Merged agent PRs drew verified follow-up fixes at 1.62 times the odds of merged human PRs from the same repositories over the same period, per the authors.
The agents mostly repair their own work: 69.6% of verified fixes to agent merges came from the same agent, and 76.4% of verified fix PRs were agent-authored across all their commits.
Candidate fixes were checked by human annotators plus an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge whose agreement with humans (Cohen's kappa 0.78) roughly matched human-to-human agreement (0.77), with 90% direct-fix precision.
The sample comes from the AIDev-pop dataset, limited to open-source repositories with at least 500 stars, so findings may not carry over to private or smaller codebases.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan
articleHow Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsHuiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
articleDocumentation Drift in AI-Generated Pull RequestsWill De Michele, Kla Tantithamthavorn