Intel / article

When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

Source: arxiv.org
Author: Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
Date: Published Sep 28, 2026

Context

Adding AI reviewers can produce an elaborate check that still cannot distinguish correct code from a buggy alternative. In this study, the MARCH framework formed claims about each solution and passed them to a separate checking . The authors found that on 76.6% of comparisons it asked the same questions about both candidates, leaving the comparison without distinguishing evidence. Run unmodified, it declared both solutions equally good on 78 to 95% of comparisons.

The authors added a gate that withholds those comparisons before the checker runs: the judge declines when it cannot separate the candidates. Accuracy over all comparisons was 20.7%, while accuracy over the roughly half it still answered was 36.9%. This measures a trade between coverage and accuracy, and the authors say the contribution is not a more accurate judge: the same model asked directly reached 43.7% where unmodified MARCH reached 4.4%. What the gate adds is a detectable reason to abstain before presenting an unsupported verdict.

Key takeaways · AI-distilled

Run unmodified on two code-judging benchmarks, the MARCH verification framework called both candidate solutions equally good in 78% to 95% of comparisons, reaching 4.4% accuracy where the same model asked directly scored 43.7%.
The authors argue claim-by-claim verification needs evidence that is independent of the answer and differs between the two candidates, a condition retrieved documents satisfy but code judging does not.
Neither easier problems nor a larger judge model changed the result; the authors explain the collapse with two label-free measurements taken from the pipeline's own logs.
Gating on a label-free signal from the pipeline's own logs let the judge decline comparisons it could not ground, raising accuracy from 20.7% to 36.9% while still answering half of all comparisons.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.

Read the source arxiv.org