Failure analysis: We classified 1,567 failing AA-AnalystAgent attempts across te

Announcing AA-AnalystAgent, our new agentic benchmark for quantitative analysis on real-world spreadsheets & documents. Claude Opus 5 leads at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49% In real analyst roles, professional judgment and expertise are as important raw quantitative capabilities. AA-AnalystAgent tests this and requires models to interpret sources, decide which exceptions and caveats apply, and settle on a methodology to successfully complete a task. Because handing analyst tasks to agents requires not just correct answers but consistent ones, AA-AnalystAgent runs each task five times and reports pass^5 as its headline metric (we call this ‘pass-all-5’). pass^5 means models must get a task correct every time it tries across five independent attempts to pass. AA-AnalystAgent overview: 🔢 80 questions across 14 business and scientific domains, including healthcare expenditure reports, trade and commodity statistics, hydrology and weather data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules 🧠 Five workflow buckets from across real analyst work: source lookup and diagnosis, filter and total, ratios/trends/sensitivities, P&L modeling, and cash/balance sheet/valuation modeling Methodology details: 🤖 Agentic harness: each task is solved by an agent running in our open-source Stirrup reference harness, with tools for code execution, web fetch, image viewing, and answer submission 📊 Scoring: each task is run five times per model, with final answers compared to reference solutions by an equality checker. Three metrics: pass^5, pass@1 (average pass rate), pass@5 🔒 Privately-held question set to limit contamination risk. Two example questions from California Medicaid expenditure reports are publicly shown on the methodology page with full prompt and source material Key findings: 🥇 @AnthropicAI's Claude Opus 5 (max) leads at 54%, followed by @OpenAI's GPT-5.5 (xhigh) at 50% and Claude Fable 5 (max, Opus 4.8 Fallback) at 49%. Anthropic holds three of the top five places, and the top three are separated by three net tasks out of 80. 🎯 Reliability separates the top of the leaderboard more than raw capability: GPT-5.5 (xhigh) has the highest pass@1, but Opus 5 leads on pass^5 because it repeats what it gets right. 🔍 Committing early to a wrong interpretation is the most widespread way models fail, appearing in 57% of the failures we classified. 💰 The price of a given score varies enormously: Claude Sonnet 4.6 and @Xiaomi's MiMo-V2.5-Pro both score 20%, at $1.34 and $0.05 per task. 🥈 @Kimi_Moonshot's Kimi K3 (max) is the top open weights model at 39%, 15 points behind the closed frontier. AA-AnalystAgent launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index. See below for further detail ⬇️

Failure analysis: We classified 1,567 failing AA-AnalystAgent attempts across ten leading models into seven failure modes, tagging each attempt with every mode it exhibited. The most widespread is anchoring on a wrong early hypothesis, present in 57% of failures. Models commit to a source or interpretation early and defend it for the rest of the trajectory. The sharpest contrasts are in how models treat sources. Gemini 3.1 Pro Preview takes sources at their word but struggles with execution. It sits below the ten-model median on both misreading domain terms and overriding evidence in favor of a generic prior, while posting above-median rates of modeling, scaling and aggregation errors (51% of its failures) and skipped verification (39%). @spacexai's Grok 4.5 (high) is the reverse. It understands the field's language, then substitutes its own assumptions for what the documents say, misreading domain terms on 23% of its failures against a 38% median, while overriding evidence on 54% against a 43% median. Kimi K3 (max) misreads on 48%, 25 points clear of Grok 4.5.