Who Changed the AI Agent's Rules? Measuring How AI Coding Agents Escalate Authority in CI/CD Pipelines
- Source
- arxiv.org
- Author
- Miroslaw Staron
- Date

Context
Miroslaw Staron's paper, accepted at the PROFES 2026 software-process conference, measures how often AI coding agents change the machinery that reviews and ships code rather than the code itself. Staron calls that machinery the control plane: CI/CD workflow files, approval gates, permissions, secrets and deployment rules. A change there alters the conditions every later pull request faces, so an that edits it is changing rules that also constrain its own work. The study scores all 932,791 pull requests in AIDev, a public dataset of pull requests opened by five agents: OpenAI's Codex, GitHub Copilot, Devin, Cursor and Anthropic's Claude Code.
About 4% of those pull requests, 37,409, were classified as control-plane changes. The classifier reads only each pull request's title and description, matching terms such as .github/workflows or branch protection, because file-level data covers only part of the dataset. On 34 pull requests the author labelled by hand, it agreed 94.1% of the time (Cohen's kappa 0.72). The control-plane share differed widely by agent, from 2.1% for Codex to 25.2% for Copilot and 27.1% for Claude Code. Reviewers merged control-plane pull requests less often than others, 73.2% against 85.2%.
The headline risk figures come from a penalty score the paper defines, and much of the gap is built into that score. It combines execution cost, an effort estimate from the description plus the time to merge or close, with governance risk made of three yes-or-no flags: closed without merging, merged despite failure signals, and a policy-violation flag that applies only to control-plane changes and is set when one merged in under five minutes or despite failure signals. Control-plane changes also receive a fixed 0.25 premium. On that score, control-plane pull requests average 0.498 against 0.218 for the rest, and their governance risk 0.530 against 0.135. All 775 pull requests above the paper's extreme-risk line of 0.8 are control-plane changes, but that follows largely from the formula: a data-plane change's governance risk is capped at one third, and the paper itself notes that under its model only control-plane changes can reach that range. The 63.6% policy-violation rate among merged control-plane changes likewise counts fast or failing merges, a proxy the author acknowledges is not a direct observation of review quality.
Mean penalty by agent ranged from 0.166 for Cursor to 0.328 for Copilot, and the author argues that authority policies should be for each agent. The study cannot say whether any flagged change actually weakened a pipeline or caused harm: its weights and the 0.25 premium have not been validated in industry, and the dataset is dominated by Codex pull requests in open-source projects. An earlier analysis drawing on the same dataset, Not All Agents Are Equal, measured outcomes rather than proxies and found that revert rates differed by vendor: 6.1% for Codex and 14.5% for Devin, against 11.5% for human pull requests. What Staron's paper does establish is that agent pull requests whose descriptions point to CI/CD configuration are common, tens of thousands in one public dataset, and that is the basis for its main recommendation: classify every agent pull request by scope and require human approval for control-plane changes.
- Across all 932,791 agent-authored pull requests in the AIDev dataset, PRs that changed CI/CD control-plane structures such as workflows and permissions had 3.9x the mean governance risk of data-plane changes (0.530 vs 0.135), the study reports.
- All 775 extreme-risk pull requests in the study modified the control plane. Policy-violation flags appeared in 46.6% of control-plane PRs and in 63.6% of the control-plane PRs that were merged.
- Mean joint penalty varied by agent, from 0.166 for Cursor to 0.328 for Copilot, and execution cost or merge rate did not explain the gap, so the author argues authority policies should be set per agent. The scope classifier scored Cohen's kappa 0.72 on 34 hand-checked PRs.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan
articleHow Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsHuiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
articleA Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent AdoptionYegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer, Brando Miranda, Bogdan Vasilescu, Sanmi Koyejo
Checking sign-in…
Loading comments…