How Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull Requests
- Source
- arxiv.org
- Author
- Huiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
- Date

Context
A study posted to arXiv on October 2, 2026 compares 1,130 performance-focused pull requests written by autonomous coding agents with 1,130 written by humans on open-source projects, collected through June 2026. According to the abstract, maintainers merged PRs less often (54.4% versus 73.3%), though the two groups used similar optimization strategies and reported validation at comparable rates.
They differ in evidence. Agent PRs relied more on static reasoning, arguing from the code that a change should be faster, and less on benchmarks: 46.1% used benchmarks against 57.2% for human PRs, although use converged by the end of the study period. In both groups about half of validated PRs reported no quantitative performance metric.
The abstract does not show whether the optimizations made software faster. The authors frame that as the open challenge and argue review criteria should check intended effects and costs, not only whether validation is present.
- Agent-authored performance PRs leaned more on static reasoning than measurement: 46.1% used benchmarks versus 57.2% of human PRs, though the authors say benchmark use converged by the end of the study period.
- Agents reported validation for code-smell refactorings as often as for resource-targeting changes, while human authors validated refactorings less often than changes aimed at resources.
- Across both agent and human PRs, about half of the PRs that reported validation included no quantitative performance metric, and trade-offs across performance objectives were rarely quantified.
- The authors recommend review criteria that check whether a PR measures its intended effect and its costs, rather than treating the mere presence of validation as sufficient. The sample is 1,130 PRs per group, collected through June 2026.
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan
articleWho Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull RequestsWannita Takerngsaksiri, Nhat Duong, Scott Barnett
articleFrom Cosmetic to Structural: A Paired Experiment on Metric-Guided Agentic RefactoringNadim Hagatulah, Markus Borg, Emma S\"oderberg
Checking sign-in…
Loading comments…