The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → is real work, not exam questions: 160 private tasks across 10 simulated worlds, each with an accounting system plus spreadsheets and PDFs. Working accountants authored the tasks, solved them, and wrote the grading rubrics.
Averages hide the reliability gap. The leading model scores 56.4% on mean criteria over three runs, but no model clears 2.6% at Pass^8, which requires getting the same task right in all eight attempts.
Even the generous framing is grim: the best Pass@8, meaning at least one of eight attempts is correct, is 21.5%. Roughly four fifths of real accounting tasks are unsolved even when the model gets eight tries.
Raising the token budgetA cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.Full definition → from $1 to $50 lifts scores, yet inside any fixed budget the tasks where the model spends the most tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → score worst. Spending more helps; needing to spend more is a signal the model is lost.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Held-out, real-work benchmarks are a better read on AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → capability than saturated academic suites.
Key quotes
“We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants.”
“Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%.”
“No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)).”
“We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens.”
articleBuild and run your own benchmark today:ArtificialAnlys
articleREAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production UsageSmriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra