Tools nearly double VIALS-AA scores and let Haiku beat Fable
- Source
- x.com
- Date
Take away its tools, and Claude Fable 5.2 scores below Claude Haiku 5.5 with tools on VIALS-AA; across models tool use nearly doubles scores, from ~25% to ~47% on average VIALS-AA is our implementation of @joinHandshakeAI's VIALS benchmark for visual reasoning in life sciences research. By default, models work as agents: they think, run code and view images in a sandbox. We reran VIALS-AA in a one-shot format: one request with the question and images, and no tools. Lower cost models in an agent harness can beat single-shot large models on both cost and performance in visual reasoning and interpretation of scientific artifacts. Claude Fable 5.2 (max) leads VIALS-AA at 67% with tools, at a cost of $4.20 per task; without tools it scores 44% at $0.73 per task. Claude Haiku 5.5 (max) scores 49% with tools, at $0.07 per task: higher than Fable without tools, at about a tenth of the cost.
Shows that a cheap model in an agent harness can beat a frontier model used one-shot on visual reasoning at about a tenth of the cost. Tools matter most for counting tasks, so harness design deserves as much as model choice.
- Artificial Analysis normally runs VIALS-AA, its version of Handshake's life-science visual reasoning benchmark, as an that thinks, runs code and views images in a . For this comparison it reran it as one request with the question and images and no tools.
- Across the models tested, tool access nearly doubled average scores, from about 25% one-shot to about 47% in the .
- Claude Fable 5.2 (max) leads at 67% with tools for $4.20 per task and drops to 44% at $0.73 without them; Claude Haiku 5.5 (max) scores 49% with tools at $0.07 per task.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
Checking sign-in…
Loading comments…