Intel / article

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

Source: arxiv.org
Author: Obada Kraishan
Date: Published Sep 17, 2026

From the Daily Brief

A study examined 37,623 pull requests with verified provenance across five autonomous coding agents, Codex, Devin, GitHub Copilot, Cursor, and Claude Code, plus a human baseline, measuring code quality, revert rates, and security smells after merge. The headline finding is that these outcomes differ sharply by vendor rather than clustering together as a single AI-generated-code category, meaning the agent a team chooses has a measurable, vendor-specific effect on downstream maintenance burden.

That undercuts a common simplification in engineering discussions: that AI-written code carries one uniform risk profile worth a single policy. Teams setting code review standards, security gates, or agent-adoption policy should treat vendor choice as a variable that actually shows up in post-merge outcomes, not just up-front task success rate. It also gives teams a basis for auditing which agents they currently allow, since revert rates and security smells are the costs that show up weeks after a PR merges, not at merge time.

Read the 2026-09-18 Brief →

Context

A study tracked what happens after an AI coding 's pull request gets merged, analyzing 37,623 provenance-labeled pull requests from five commercial agents, OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code, against a matched human baseline across 2,807 GitHub repositories from December 2024 through July 2025 (arxiv.org). Quality differences turned out to be vendor-specific rather than a single 'AI code' pattern: Codex-authored PRs were reverted about half as often as human PRs (6.1 percent versus 11.5 percent), while Devin PRs were reverted more often, at 14.5 percent. Pooled across all five vendors, agent code was less likely than human code to contain a security smell, largely because it had fewer hardcoded credentials and -style constructs. Human review effort concentrated unevenly too: Copilot PRs drew the most reviews and change requests, while Claude Code PRs waited the longest for a first human review, a median of 12.6 hours.

Key takeaways · AI-distilled

Codex-authored PRs were reverted about half as often as human-authored PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often than human code (14.5%, odds ratio 1.31) — quality moved in opposite directions by vendor.
Pooled across all five agents, agent-written code was less likely than human code to contain a security smell (odds ratio 0.63), driven specifically by fewer hardcoded credentials and eval-style constructs rather than a uniform improvement across all smell types.
Human review is uneven across agents: Copilot PRs drew the most human reviews and change requests, while Claude Code PRs waited the longest for a first human review, a median of 12.6 hours.
The dataset spans 37,623 provenance-labeled PRs across 2,807 GitHub repositories from December 2024 to July 2025, combining the AIDev dataset with 58,792 cached GitHub API responses, with all pipeline code and statistical reports released for replication.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Read the source arxiv.org