Frontier models reach only about 70% on realistic 100k-token multi-document reasoning, and non-reasoning long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → models can beat reasoning ones. Use this to pick models for large codebase or document work.
Key takeaways · AI-distilled
Human evaluators working from the same documents typically answered only 40-60% of AA-LCR questions correctly on a first attempt, though every question was answered correctly by at least one human tester.
Question writers checked difficulty against non-frontier models (GPT-4o-mini, Llama-3.1-70B, Gemini 1.5 Flash) so the questions would not be adversarially tuned against any particular frontier model.
Each question needs both multi-document and multi-step inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →, with scenarios such as comparing financial metrics across company reports, tracking entities in legal documents and synthesizing positions from government consultations.
Artificial Analysis reports large differences in output-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → use: 2.7 million for OpenAI o3 versus 22,000 for Amazon Nova Premier. These are token counts, not a comparison of monetary cost.
The published article's Sample Question section ends with an incomplete code fence and supplies no example. Its methodology and result descriptions are available, but the missing example cannot be checked from this page.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
articleAre the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon StatementsXinke Tong, Xuanming Zhang, Tianyi Tang, An Yang, Jiatu Hu, Guojie Lin, Zhenzhen Shi, Lingfeng Zeng, Boyu Yang, Bing Zhao, Hu Wei, Lin Qu, Dayiheng Liu
postLong medical case files as a reasoning benchmarkArtificial Analysis