Intel / article

ExplainBench: Evaluating Code Explanations from Agents

Source: arxiv.org
Author: Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury
Date: Published Jul 30, 2026

Context

A coding 's ability to fix software does not establish how accurately it explains the change. ExplainBench evaluates explanations by asking another questions about the buggy program's intended behavior and the patch's effects. The authors found that this measure ranked agents differently from and exposed explanations that claimed incorrect patches were correct. An additional auditing agent ran tests and revised explanations, improving measured explanation quality for every evaluated agent. The benchmark measures usefulness through model question-answering, rather than directly testing human understanding, but makes explanation accuracy a separate object of evaluation. Research abstract.

Key takeaways · AI-distilled

ExplainBench scores an explanation by whether an LLM can answer questions about the code using that explanation alone. Explanation quality becomes a number instead of a vibe.
Explaining well is a separate from patching well: the benchmark ranks coding agents differently than SWE-bench Verified. The agent that fixes the most bugs is not necessarily the one that tells you the truth about its fix.
The most common failure is the most dangerous one. Agents routinely claim a patch is correct when it is not, and the explanation reads just as confident either way.
An audit agent that runs extra tests against the patch and then revises the explanation improved every agent tested, so trustworthiness can be bolted on afterward rather than trained in.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.

Key quotes

“This makes manual review of agent results increasingly infeasible, leading developers to turn to explanations to understand enacted changes.”

“Experiments first reveal that explanation quality is a distinct axis of agent evaluation: ExplainBench ranks agents differently from the widely-used SWE-bench Verified benchmark.”

“A deeper breakdown of explanation quality in agents shows frequent problems in explanations, such that explanations often claim that a patch is correct when it is not.”

“This agent improved the explanations of all evaluated agents, demonstrating agent explanations can be automatically made more trustworthy.”

Read the source arxiv.org