Sonnet 5.5 nears Opus 5.5 on agentic evals but burns record output tokens
- Source
- x.com
- Date
This is a follow-up to yesterday's launch coverage, which reported Anthropic's claim that Sonnet 5.5 finishes tasks with fewer tokens. Artificial Analysis's independent run found the opposite pattern on its suite. Sonnet 5.5 scored 56 on the Intelligence Index and reached parity with Opus 5.5 on several agentic evaluations, but used roughly 60% more output tokens than Sonnet 5, the most the firm says it has measured. That puts cost per task at $7.60, about 50% higher than Sonnet 5 at the same list price.
It repeats the pattern Artificial Analysis found for Opus 5.5 earlier this week, where extra token use offset a price cut. The two measurements can both hold, since token use varies by task and by reasoning setting. Anthropic has published migration and tuning guidance for Sonnet 5.5, and the practical advice is the same either way: compare cost per completed task on your own workload, not the per-token price.
Anthropic has launched Claude Sonnet 5.5: it scores 56 on the Artificial Analysis Intelligence Index, just 2 points behind Opus 5.5 (max), but at the highest Output Tokens per Task we’ve seen With max effort, Sonnet 5.5 gains 18 points over Sonnet 5 and to #2 on the Intelligence Index behind only Opus 5.5 (max). Anthropic has priced Sonnet 5.5 identically to Sonnet 5 at $0.2/$2/$10 per 1M cache input/input/output tokens, however it outputs a higher number of Output Tokens per Task and costs $7.60 per task (~50% higher than Sonnet 5’s Cost per Task) Key takeaways: ➤ Meets leading models on agentic terminal use and knowledge work: in Terminal-Bench 4.0, Claude Sonnet 5.5 reaches 64% against 60% for Opus 5.5 and GPT-6 Astra. On AA-Briefcase (1811 vs 1822 Elo), GDPval-AA (1844 vs 1846 Elo), and AutomationBench-AA (71% vs 70% headline score), Sonnet 5.5 reaches parity with Opus 5.5, albeit with significantly higher token usage to achieve it ➤ Heaviest token use we have measured: at max effort, where it reaches performance nearing that of Opus 5.5, Claude Sonnet 5.5 used ~193k Output Tokens per Intelligence Index Task. This is the highest token use we have measured on around 60%…
Context
Artificial Analysis, which runs its own benchmark evaluations of AI models, found in its launch evaluation that Anthropic's Claude Sonnet 5.5 comes close to the larger Opus 5.5 on its tests, but only by generating far more text per task, so a price unchanged from Sonnet 5 does not mean an unchanged bill. At max effort, the highest of the model's five effort settings, which control how much work the model puts into a problem before answering, Sonnet 5.5 scored 56 on the firm's Intelligence Index. That is 18 points above Sonnet 5 and 2 points behind Opus 5.5 at max effort, good for second place on the index.
The closeness shows up most on -style work. On Terminal-Bench 4.0, a test of agentic work in a command-line terminal, Sonnet 5.5 reached 64% against 60% for both Opus 5.5 and OpenAI's GPT-6 Astra. On three of the firm's knowledge-work evaluations it effectively tied Opus 5.5: a rating of 1811 against 1822 on AA-Briefcase, 1844 against 1846 on GDPval-AA, and 71% against 70% on AutomationBench-AA. It still trails Opus 5.5 on factual knowledge, scoring 54% against 66% for factual accuracy on AA-Omniscience, though with a lower rate, 47% against 59%, and it sits about 6 points lower on Humanity's Last Exam and SciCode.
The cost comes from output tokens, the units of text a model generates and is billed for. Anthropic kept Sonnet 5.5 at Sonnet 5's price of $2 per million input tokens and $10 per million output tokens, but at max effort the model used about 193,000 output tokens per Intelligence Index task. Artificial Analysis says that is the highest it has measured, around 60% more than Opus 5.5 or Sonnet 5 at max effort and about seven times GPT-6 Astra. The result is a cost of $7.60 per task, roughly 50% above Sonnet 5. On the firm's comparison of intelligence against cost per task, Sonnet 5.5 sits off the frontier of best value: at its higher effort levels it falls behind Opus 5.5, and at lower ones GPT-6 Astra or GPT-6 Sol configurations deliver equivalent performance for less. Its high setting comes closest, very narrowly behind GPT-6 Sol on the index at effectively the same cost per task.
These numbers come from a pre-release deployment. Anthropic found a bug in it that can degrade responses to requests using structured outputs, a mode in which the model must answer in a fixed machine-readable format. The bug is fixed in the public release, Anthropic expects minimal change or slightly better results, and Artificial Analysis says it will rerun the relevant evaluations, so individual scores may still move. A later Artificial Analysis post compared Sonnet 5.5's effort levels on the firm's Coding Agent Index.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
Checking sign-in…
Loading comments…