ExplainBench: Evaluating Code Explanations from Agents
Source
Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury
Author
Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury
Date
Key takeaways · AI-distilled
ExplainBench scores an explanation by whether an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → can answer questions about the code using that explanation alone. Explanation quality becomes a number instead of a vibe.
Explaining well is a separate agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → from patching well: the benchmark ranks coding agents differently than SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition →. The AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → that fixes the most bugs is not necessarily the one that tells you the truth about its fix.
The most common failure is the most dangerous one. Agents routinely claim a patch is correct when it is not, and the explanation reads just as confident either way.
An audit agent that runs extra tests against the patch and then revises the explanation improved every agent tested, so trustworthiness can be bolted on afterward rather than trained in.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Why it matters
When agent diffs outgrow manual review, the explanation is the review — and it needs to be measured.
Key quotes
“This makes manual review of agent results increasingly infeasible, leading developers to turn to explanations to understand enacted changes.”
“Experiments first reveal that explanation quality is a distinct axis of agent evaluation: ExplainBench ranks agents differently from the widely-used SWE-bench Verified benchmark.”
“A deeper breakdown of explanation quality in agents shows frequent problems in explanations, such that explanations often claim that a patch is correct when it is not.”
“This agent improved the explanations of all evaluated agents, demonstrating agent explanations can be automatically made more trustworthy.”
articleREAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production UsageSmriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra
articleSearchAuditor: Auditing and Attributing Failures in Long-Horizon Search AgentsZhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
articleWikiBench: does a generated codebase wiki actually help a coding agent?LangChain