Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Source
arxiv.org
Author
Md Shamimur Rahman, Khairul Alam, Banani Roy, Chanchal K. Roy
Date
Why it matters
Reviewers hold AI-authored PRs to a higher evidence bar and lean on test inspection and local execution rather than author track record. If you submit agent PRs, include evidence reviewers can verify.
Key takeaways · AI-distilled
For AI-authored PRs, surveyed reviewers relied less on author track record and subsystem familiarity, and more on the expertise of the human who prompted the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → and on keeping the change small.
55.2% said their own reasoning through critical cases was enough to approve an AI-authored PR, versus 28.5% for human PRs, and they leaned less on test coverage for AI PRs.
The share who would block approval under residual uncertainty rose from 19.4% for human PRs to 43.9% for AI PRs, and unclear ownership paused AI PR review more often (29.3% versus 14.6%).
The authors recommend review tools and agent workflows expose provenance, separate generated claims from human-validated evidence, and make uncertainty visible before approval.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleHow Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsHuiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
articleGroundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding AgentsJunyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
articleHow Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific ProgrammingGabrielle O'Brien, Reed Milewicz, Nasir Eisty