Intel / article

APEX-Accounting

Source: arxiv.org
Author: Julien Benchek, Austin Bennett, Jasmin Kern, Ryan Stevens, Rene Sultan, Charis Ching, Hayley Popiel, Vaibhav Mittal, Felix Mercier, Brendan Foody, Bertie Vidgen
Date: Published Jul 30, 2026

Context

APEX-Accounting shows how partial progress can conceal unreliable completion. Built by Mercor with Ramp, its private contains 160 expert-authored tasks across ten synthetic company environments, with files and accounting tools. Across nine models, the highest score was 56.4% of grading criteria satisfied (Claude-Fable-5 at its Max setting), averaged over three runs. That measures partial credit within tasks. Benchmark and scoring method.

Full completion was much harder. The highest Pass@8, completing a task at least once in eight attempts, was 21.5% (Muse-Spark-1.1, xHigh); the highest Pass^8, completing it correctly in all eight, was 2.6% (GPT-5.6-Sol, Max+Pro). Those separate measures expose the difference between occasionally producing a complete answer and doing so consistently. Reliability results.

Key takeaways · AI-distilled

The is real work, not exam questions: 160 private tasks across 10 simulated worlds, each with an accounting system plus spreadsheets and PDFs. Working accountants authored the tasks, solved them, and wrote the grading rubrics.
Averages hide the reliability gap. The leading model scores 56.4% on mean criteria over three runs, but no model clears 2.6% at Pass^8, which requires getting the same task right in all eight attempts.
Even the generous framing is grim: the best Pass@8, meaning at least one of eight attempts is correct, is 21.5%. Roughly four fifths of real accounting tasks are unsolved even when the model gets eight tries.
Raising the from $1 to $50 lifts scores, yet inside any fixed budget the tasks where the model spends the most score worst. Spending more helps; needing to spend more is a signal the model is lost.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Key quotes

“We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants.”

“Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%.”

“No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)).”

“We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens.”

Read the source arxiv.org