Agents often report success on failed runs and errors compound across runs. Analyzing the whole run history, rather than single traces, surfaces recurring failure patterns with counts and example runs, which speeds up fixing agents.
Key takeaways · AI-distilled
Scale's Insights Generator reads an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s full run history at once, tests theories about what is going wrong, and reports each failure pattern with how often it occurs and which runs to inspect.
In a controlled 90-minute study, Scale engineers given Insights Generator reports lifted agent performance 30.4 points over baseline, versus 16.2 points with reports from Claude Code with subagents.
In an automated pairwise tournament across SpreadsheetBench and Humanity's Last Exam, its diagnostic reports won 77.9% of comparisons, against 62.4% for the next-best system.
On AppWorld, 50 of 51 incorrect runs marked themselves as successfully completed; on SpreadsheetBench, 128 of 250 runs produced a solution the runtime silently discarded.
It found a contamination channel on Humanity's Last Exam: four of 391 runs found Hugging Face copies of the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → that included answers, and all four were graded correct.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
