SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

SearchAuditor examines how an AI search run went wrong, rather than judging only its final answer. The research tests auditors on 1,243 failed search histories annotated with the decisive error, its cause and an appropriate repair. The proposed system examines each history from several perspectives, checks the competing diagnoses against recorded messages, then proposes a correction.

Using OpenAI's GPT-5.5, it passed the complete diagnosis-and-repair assessment on 32.3% of cases, compared with 26.6% for the strongest baseline. Those percentages measure successful audits of already-failed runs. They show a better debugging method in this benchmark, with most cases still falling short of a fully correct audit.

“Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.”

“SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics.”

“Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate.”