Intel / article

Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report

Source: arxiv.org
Author: Eric Jansson, Per Strandberg, Thomas S\"orensen, Eduard Paul Enoiu, Wasif Afzal
Date: Published Sep 21, 2026

Context

Using several AI agents did not consistently improve reports explaining failed software tests in this industrial case study. Researchers compared one with an orchestrated team, giving both access to test metadata and logs. Six practitioners assessed reports for two real failure scenarios, while 120 repeated executions supplied operational measurements.

The authors report that the single agent produced reports faster and at lower cost, with neither configuration earning a consistent quality advantage from reviewers. That makes one agent the better-supported baseline for this particular workflow. The study's narrow set of failure scenarios leaves open whether more complex investigations would benefit from a team.

Key takeaways · AI-distilled

The study ran two real failure scenarios through both single-agent and orchestrated RCA configurations, each with access to the same test metadata and logs, then measured 120 repeated executions for cost, duration, and consistency.
Six practitioners rated the resulting reports via survey and focus group on six axes: correctness, reasoning quality, fix realism, clarity, usefulness, and trust.
Despite the added architecture, multi-agent showed no consistent practitioner-perceived quality edge across the two scenarios tested, while the single-agent system was both faster and cheaper to run.
The authors' conclusion is qualified, not dismissive: they say agent-architecture benefits require further in more complex scenarios, not that multi-agent RCA doesn't work.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source arxiv.org