Announcing AA-AnalystAgent, our new agentic benchmark for quantitative analysis
- Source
- x.com
- Date
Announcing AA-AnalystAgent, our new agentic benchmark for quantitative analysis on real-world spreadsheets & documents. Claude Opus 5 leads at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49% In real analyst roles, professional judgment and expertise are as important raw quantitative capabilities. AA-AnalystAgent tests this and requires models to interpret sources, decide which exceptions and caveats apply, and settle on a methodology to successfully complete a task. Because handing analyst tasks to agents requires not just correct answers but consistent ones, AA-AnalystAgent runs each task five times and reports pass^5 as its headline metric (we call this ‘pass-all-5’). pass^5 means models must get a task correct every time it tries across five independent attempts to pass. AA-AnalystAgent overview: 🔢 80 questions across 14 business and scientific domains, including healthcare expenditure reports, trade and commodity statistics, hydrology and weather data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules 🧠 Five workflow buckets from across real analyst work: source lookup and diagnosis, filter and…
Context
AA-AnalystAgent, a new from Artificial Analysis, scores AI models on whether they get an analyst task right every time across repeated attempts. Each of its 80 tasks asks an to work through real spreadsheets and documents from 14 business and scientific domains, such as healthcare expenditure reports, government appropriations and financial models, and then decide which caveats apply and which method to use. Every model runs each task five times, and the headline metric, pass^5, counts a task only if all five attempts match the reference answer. The group also reports two other metrics, pass@1, the average pass rate, and pass@5. The announcement explains the choice by saying that handing analyst work to agents needs consistent answers as well as correct ones.
That rule changes the order at the top. Anthropic's Claude Opus 5 (max) leads at 54%, ahead of OpenAI's GPT-5.5 (xhigh) at 50%, even though GPT-5.5 has the higher average pass rate; Artificial Analysis attributes the difference to Opus 5 repeating the answers it gets right. The margins are small. The top three models are separated by three net tasks out of 80, so a few tasks going the other way could reorder them. Lower down, the price of reaching the same score varies enormously: Claude Sonnet 4.6 and Xiaomi's MiMo-V2.5-Pro both score 20%, at $1.34 and $0.05 per task. Kimi K3 (max) is the best open-weights model at 39%.
The question set is kept private to limit the chance that models were trained on it, and the agents run in Artificial Analysis's open-source Stirrup with tools for running code, fetching web pages and viewing images. The benchmark is a standalone leaderboard and does not feed the Artificial Analysis Intelligence Index. The same thread includes a failure analysis, which found that committing early to a wrong interpretation appeared in 57% of classified failures.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Checking sign-in…
Loading comments…