Human annotation of 204 failed OSWorld trajectories puts task reasoning and control as the largest computer-use-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → failure family (110/204), ahead of perception (36) and groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition →/interaction (25).
CUADebugger actively inspects suspicious steps with paired before/after screenshots instead of prompting once over the whole trajectory, emitting a structured root-cause diagnosis with corrective strategy.
Root-cause-informed re-execution nearly doubles task completion vs history-only continuation (29.9% vs 13.9% single-shot; 12.2% to 25.9% continual, near the 29.2% human-oracle ceiling).
The takeaway for agent builders: diagnosis is an actionable repair signal, not just a post-hoc explanation - and it transfers across debugger backbones.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
It gives builders of computer-use agents a concrete error taxonomy, a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → to measure failure modes, and a debugging approach that roughly doubles task success versus naive retries — a rare evidence-backed fix for a widely acknowledged pain point.
Key quotes
“CUA failures arise from coupled visual perception, spatial grounding, low-level interaction, task reasoning, and environment dynamics, making debugging a distinctive multimodal causal localization problem.”
“These results show that CUA root-cause diagnosis can provide actionable repair signals rather than merely post-hoc explanations.”
articleUnderstanding Bugs in Modern Agentic Frameworks: A Study of Symptoms, Root Causes, and Triggering ConditionsXiaowen Zhang, Hannuo Zhang, Shin Hwei Tan
articleSearchAuditor: Auditing and Attributing Failures in Long-Horizon Search AgentsZhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
articleFantastic Adaptive Taxonomies and How to Use ThemMert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica