Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica
Author
Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica
Date
Key takeaways · AI-distilled
AdaMAST reads an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → system's own traces and induces a failure taxonomy: named failure codes organized on three fixed axes — system-level, role-specific, domain-specific. No code is hand-authored and no trace is human-annotated.
Coded failure feedback beats free-text self-reflection at runtime: SWE-agent goes from 60% to 70% on SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Mini, and Claude Code from 64.0% to 70.7% when the taxonomy is supplied as a runtime agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →.
A verifier built on the induced codes picks among five candidate trajectories and lifts Terminal-Bench 2.0 accuracy 8-15 points over a single attempt, so the vocabulary works as a selection signal, not just a post-hoc diagnosis.
Taxonomies induced for different domains share few codes, so this is not portable boilerplate — each system needs its own. The compressed code set still matched expert failure annotations more closely than a hand-crafted reference vocabulary.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
Why it matters
Every weight-free agent improvement loop reads traces; giving those traces a stable structure is what makes the feedback compound.
Key quotes
“We argue that an agent system should instead maintain an explicit representation of how it fails, induced from its own behavior and reusable wherever failure feedback is needed.”
“At runtime, taxonomy feedback raises SWE-agent's resolution on SWE-bench Verified Mini from 60% with free-text reflection to 70%, and improves Claude Code from 64.0% to 70.7% as a runtime skill.”
“In trajectory selection, AdaMAST-Judge, a verifier built on the induced codes, improves best-of-5 accuracy on Terminal-Bench 2.0 by 8-15 points over Pass@1.”
“Adaptive failure taxonomies close the loop between the traces agents produce and the procedures that improve them.”
articleSearchAuditor: Auditing and Attributing Failures in Long-Horizon Search AgentsZhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
postHow to really stop your agents from making the same mistakesGarry Tan
articleOpenDiscoveryTrace: Process Traces for Evaluating AI Scientist WorkflowsAayam Bansal, Keertan Balaji