Llm Apis Accuracy Runtime Tradeoff

This June 11, 2025 data insight from Epoch AI, by Greg Burnham and Tom Adamczewski, puts a number on the trade-off between accuracy and speed. Using its own benchmark runs, Epoch plotted each model's accuracy against the average time it took to answer a question on three question-and-answer tests: GPQA Diamond, MATH Level 5 and OTIS Mock AIME 2024-2025. Each answer is a single API call with no agent scaffolding, so runtime here is simply how long the model takes to respond. Epoch then looked only at the Pareto frontier, the models that are faster than every more accurate model or more accurate than every faster one.

Along that frontier, cutting the error rate in half was associated with a 6.0x increase in runtime on GPQA Diamond, 1.7x on MATH Level 5 and 2.8x on OTIS Mock AIME. Epoch found the relationship held regardless of starting point: a model with twice the runtime cut its error by about the same factor whether it was fast or slow to begin with. The frontier itself was dominated by OpenAI and Google, which supplied 15 of the 19 frontier models, and 12 of those 19 have flash, mini or nano in their names, smaller versions of larger models optimised for speed and cost.

The fits are tight but rest on few points: 12, 8 and 11 frontier models respectively. With OpenAI and Google models removed, the frontier still slopes upward, though the runtime multipliers grow, to 8.9x on GPQA Diamond. Epoch also cautions that its runs took place on different dates through 2025 and that providers change how fast they serve a model over time, an uncertainty its analysis assumes is not large enough to invalidate the finding (epoch.ai).