Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report
Source
Eric Jansson, Per Strandberg, Thomas S\"orensen, Eduard Paul Enoiu, Wasif Afzal
Author
Eric Jansson, Per Strandberg, Thomas S\"orensen, Eduard Paul Enoiu, Wasif Afzal
Date
Key takeaways · AI-distilled
The study ran two real failure scenarios through both single-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → and orchestrated multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → RCA configurations, each with access to the same test metadata and logs, then measured 120 repeated executions for cost, duration, and consistency.
Six practitioners rated the resulting reports via survey and focus group on six axes: correctness, reasoning quality, fix realism, clarity, usefulness, and trust.
Despite the added architecture, multi-agent showed no consistent practitioner-perceived quality edge across the two scenarios tested, while the single-agent system was both faster and cheaper to run.
The authors' conclusion is qualified, not dismissive: they say agent-architecture benefits require further evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → in more complex scenarios, not that multi-agent RCA doesn't work.
Terms in this piece · Glossary
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
A real production deployment found multi-agent orchestration for root-cause analysis didn't reliably beat a single agent on practitioner-judged quality while costing more and running slower, a useful data point against defaulting to multi-agent complexity.
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
articleAgentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration TestingIsrat Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu, Mohammad Sakib Mahmood, Md. Jakir Hossain, M. F. Mridha
articleCode Health in LLM-Based Test Generation: Effectiveness and Token EfficiencyFreya Wirdemann, Markus Borg, Nadim Hagatulah, Adam Tornhill