How Much Does Reasoning Effort Affect Score and Cost?

Turning a model's reasoning setting down saves far more money than it costs in score, according to a study by Vals AI, which runs the Vals Index of agentic finance, coding, legal and tax benchmarks. Most frontier models let the caller choose how much the model thinks before answering, and higher settings produce more reasoning tokens and a larger bill. Vals normally runs each model at its highest setting. Testing the top model from seven labs at three settings, it found that four of the seven kept at least 80% of their index score at their lowest setting for half the cost or less. Claude Opus 5.5 kept 80% of its score at 16% of the cost, GPT-6 Astra kept 88% at 18% and Grok 4.7 kept 86% at 16%. Gemini 4 Argon showed the opposite pattern: it kept 99% of its score but still paid 84% of the cost.

The savings are real, but so is the loss. On the index as a whole, no model matched its leaderboard score at its lowest setting, and 33 of the cheaper settings Vals tested scored significantly lower than the leaderboard setting on individual benchmarks. In 22 of the 56 model and benchmark combinations, a cheaper setting scored as high as the leaderboard setting. How much extra thinking helps depends on the work. Moving from the lowest to the highest setting gained a median 19.2 points on Legal Research and 15.6 on Code Migration, but less than 6 points on Finance Agent, EMB and HLAB.

Choosing the setting task by task does not unlock much more. Looking back at its results, Vals picked the cheapest setting that achieved the best score on each task. That hindsight choice raised scores by a median of 6.0 points over the leaderboard setting at 64% of its cost, and Vals calls it an upper bound because a real system has to choose before seeing the result. The comparisons use about half of each benchmark's tasks, usually with one run per setting, so small real differences can go undetected, and most leaderboard-setting runs date from earlier months while the other settings ran in October. For cost-sensitive work, the results favor trying a lower setting first in areas where Vals saw small gains, while legal research and code migration still rewarded paying for more reasoning. This study widens an earlier Vals test of reasoning effort that covered two models on a single benchmark.