Announcing AA-AnalystAgent, our new agentic benchmark for quantitative analysis on real-world spreadsheets & documents. Claude Opus 5 leads at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49% In real analyst roles, professional judgment and expertise are as important raw quantitative capabilities. AA-AnalystAgent tests this and requires models to interpret sources, decide which exceptions and caveats apply, and settle on a methodology to successfully complete a task. Because handing analyst tasks to agents requires not just correct answers but consistent ones, AA-AnalystAgent runs each task five times and reports pass^5 as its headline metric (we call this ‘pass-all-5’). pass^5 means models must get a task correct every time it tries across five independent attempts to pass. AA-AnalystAgent overview: 🔢 80 questions across 14 business and scientific domains, including healthcare expenditure reports, trade and commodity statistics, hydrology and weather data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules 🧠 Five workflow buckets from across real analyst work: source lookup and diagnosis, filter and total, ratios/trends/sensitivities, P&L modeling, and cash/balance sheet/valuation modeling Methodology details: 🤖 Agentic harness: each task is solved by an agent running in our open-source Stirrup reference harness, with tools for code execution, web fetch, image viewing, and answer submission 📊 Scoring: each task is run five times per model, with final answers compared to reference solutions by an equality checker. Three metrics: pass^5, pass@1 (average pass rate), pass@5 🔒 Privately-held question set to limit contamination risk. Two example questions from California Medicaid expenditure reports are publicly shown on the methodology page with full prompt and source material Key findings: 🥇 @AnthropicAI's Claude Opus 5 (max) leads at 54%, followed by @OpenAI's GPT-5.5 (xhigh) at 50% and Claude Fable 5 (max, Opus 4.8 Fallback) at 49%. Anthropic holds three of the top five places, and the top three are separated by three net tasks out of 80. 🎯 Reliability separates the top of the leaderboard more than raw capability: GPT-5.5 (xhigh) has the highest pass@1, but Opus 5 leads on pass^5 because it repeats what it gets right. 🔍 Committing early to a wrong interpretation is the most widespread way models fail, appearing in 57% of the failures we classified. 💰 The price of a given score varies enormously: Claude Sonnet 4.6 and @Xiaomi's MiMo-V2.5-Pro both score 20%, at $1.34 and $0.05 per task. 🥈 @Kimi_Moonshot's Kimi K3 (max) is the top open weights model at 39%, 15 points behind the closed frontier. AA-AnalystAgent launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index. See below for further detail ⬇️