Intel / article

Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

Source: arxiv.org
Author: Xiaonan Xu, Wenjing Wu
Date: Published Aug 3, 2026

Context

A passing test is weak evidence for a bug fix if it also passes on the broken code. This study replayed repair agents' validation commands against the original buggy code, the 's candidate, and the developer's reference fix. Across 3,730 events on 110 tasks, 46% of comparable positive events carried no information distinguishing the bug. Such tests may still check other behavior; the percentage is not a count of incorrect patches. Feedback showing the buggy-code result reduced closures with inadequate evidence by 7.8 percentage points versus a matched reminder, below the researchers' predefined threshold for practical importance.

Key takeaways · AI-distilled

The method captures each validation command, extracts a test-only patch, and replays it against three code states: the original buggy version, the agent's candidate, and the developer's real fix.
A test that passes on the candidate and also on the untouched buggy code proves nothing about the reported defect. Almost a quarter of baseline runs closed with a patch whose entire positive evidence was that kind of test.
Handing the agent the buggy-state replay result cut evidence-inadequate closures by 7.8 percentage points and raised bug-discriminating evidence by 7.4, with no measurable cost to repair success.
The honest caveat: both effects land under the 10-point threshold the authors preregistered as practically meaningful, and about a third of the gain came from the bare reminder rather than the replay content.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect.”

“46.0% of positive comparable events carry no bug-discriminating information”

“Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success.”

Read the source arxiv.org