Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents
- Source
- arxiv.org
- Author
- Junyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
- Date
Many teams now ask one model to review another model's patches, and the instinct is to make the reviewer as large as possible. This paper from Junyu Guo and coauthors suggests the instinct points at the wrong variable. Across 411 coding-agent traces, the authors measured how well reviewer models caught defective patches. Reviewer scale on its own predicted little. What mattered was groundability: reviewers that worked from execution evidence, such as tests that fail on the unpatched repository, caught more defects and rejected fewer correct patches. Reviewers relying on confident summaries did worse. The paper also describes a cascade that generates its own tests, which makes the approach usable on repositories that ship without official tests. The consequence for agent pipelines is that a cheaper reviewer can be enough, provided the harness gives it something executable to check rather than only the diff and the agent's explanation.

Context
A nominally weaker AI reviewer can assess a stronger coding more effectively when it receives decisive evidence about whether the patch works. In a study of execution-labeled agent traces, supplying official execution evidence improved both defect detection and avoidance of false rejection for five of six reviewers on 122 held-out traces.
Producing comparable evidence without official tests remained difficult. A separate review pipeline using static checks and generated tests still frequently rejected correct patches. The reported result points to the quality of the checks supplied to the reviewer as a central constraint, rather than reviewer size alone.
- On 154 GPT-5.4 traces, giving reviewers structured but unchecked evidence raised defect catch and over-rejection together, so more alone did not make review more reliable.
- With official execution evidence, five of six reviewers improved both catch and over-rejection on 122 held-out traces, and two classified every trace correctly. Reviewer size was not a consistent predictor of quality.
- A deployable cascade using patch-caused static errors and generated tests that first fail on the unpatched repo reached catch of 0.76 and 0.80, but over-rejection stayed high at 0.66 and 0.67 on GPT-5.4 and Gemini traces.
- Most false rejections happened when unresolved cases reached the reviewer. The authors name producing reliable checks without official tests as the main remaining bottleneck.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleCross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection StudyBowen Xu, Boyu Chen
articleExecuCritic: Calibrated Critic Shaping for Code Generation with Verifiable RewardsJunjie Cao, Yingjie He
articleTestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator EvolutionShuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
Checking sign-in…
Loading comments…