Vals AI: when extra reasoning effort stops paying for itself
- Source
- x.com
- Date
Vals AI introduced MysteryMechanism, a new benchmark, and swept both GPT-6 Astra and Claude Opus 5.5 across every available reasoning effort setting. Accuracy does rise with longer thinking, but the top settings cost up to 20 times as much as the lower ones for the gain they deliver.
The more interesting finding is qualitative. Inspecting the reasoning traces, Vals reports that the two models spend their extra effort in very different ways, which means an effort level that is efficient for one model is not a safe default for the other.
Latent Space's daily digest adds a data point in the same direction: Opus 5.5 scored 62% on Terminal-Bench-Science at xhigh effort and fell to 59% at max. Taken together, effort is a tuning parameter to measure per model and per task, not a dial to leave at maximum. Teams running agents at scale should record accuracy and spend at two or three effort levels on their own workload before committing.
When does more reasoning stop being useful and start becoming overhead? We ran Open AI and Anthropic’s best models: GPT-6 Astra and Claude Opus 5.5 at every reasoning level on our new benchmark, MysteryMechanism. Thinking longer delivers accuracy gains with the same model at as much as 20x the cost. Inspecting traces, the models use their effort level in very different ways.
Context
Vals AI built a new , MysteryMechanism, to test what happens as you raise a model's reasoning effort setting, the dial that trades more time and for a chance at a better answer. Running OpenAI's GPT-6 Astra and Anthropic's Claude Opus 5.5 at every effort level, Vals found that pushing a model from a lower to a higher setting can raise accuracy, but the cost of that gain can run as much as 20x more for the same model.
Vals says inspecting the models' reasoning traces at each effort level showed the two models use their extra effort in very different ways, though the post does not spell out what that difference looks like. Vals ran a similar cost-versus-effort comparison on math proofs earlier this month and found effort mattered most going from low to medium; MysteryMechanism extends that question to a different, unnamed task.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…