Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
Source
Obada Kraishan
Author
Obada Kraishan
Date
Key takeaways · AI-distilled
Codex-authored PRs were reverted about half as often as human-authored PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often than human code (14.5%, odds ratio 1.31) — quality moved in opposite directions by vendor.
Pooled across all five agents, AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →-written code was less likely than human code to contain a security smell (odds ratio 0.63), driven specifically by fewer hardcoded credentials and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →-style constructs rather than a uniform improvement across all smell types.
Human review attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → is uneven across agents: Copilot PRs drew the most human reviews and change requests, while Claude Code PRs waited the longest for a first human review, a median of 12.6 hours.
The dataset spans 37,623 provenance-labeled PRs across 2,807 GitHub repositories from December 2024 to July 2025, combining the AIDev dataset with 58,792 cached GitHub API responses, with all pipeline code and statistical reports released for replication.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Why it matters
Gives concrete, vendor-specific numbers on coding-agent code quality and post-merge maintenance instead of treating 'AI-generated code' as a single category.
articleA Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent AdoptionYegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
articleAgentLogs: A Dataset for Opening the Black Box of GitHub's Cloud AgentJonan Richards, Kosei Horikawa, Youmei Fan, Yutaro Kashiwa, Mairieli Wessel