Vibeleaderboard
← All Intel
Intel / post

Harness design cut agent cost 40% across six labs' models

Source
x.com
Date
WRITER@Get_Writer

[NEW RESEARCH] A better harness cuts AI cost and duration by 40%+, at the same quality, on any model. We benchmarked models from 6 leading labs across 22 tasks, comparing a harness optimized for token efficiency against a naive harness optimized only for completing the task. The efficient harness won on cost and speed by over 40%, with no quality tradeoff. Two findings stood out: → 𝐓𝐡𝐞 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐠𝐚𝐢𝐧 𝐢𝐬 𝐮𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥. Every model got cheaper under the optimized harness, by 33% to 61%, regardless of its size or capability. → 𝐓𝐡𝐞 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 𝐠𝐚𝐢𝐧 𝐢𝐬 𝐧𝐨𝐭. How much quality a model extracted from the same harness scaled almost perfectly with the model's baseline strength. Stronger models got proportionally more value out of harness optimization, not just better results. We call this pairing 𝐡𝐚𝐫𝐧𝐞𝐬𝐬 𝐥𝐞𝐯𝐞𝐫𝐚𝐠𝐞. This is the thinking behind Palmyra X6 and the platform updates we shipped alongside it, including the ability to choose the right model for the task, not just the biggest one. Read the full research and see what's new →

Context

WRITER reports that changing the software around an reduced cost and duration by more than 40% across six tested models and 22 tasks, with comparable assessed quality. Its engineering account describes keeping stable prompt content reusable through caching, compressing older history and limiting repeated failed attempts. These changes target what the model processes and how often it must repeat work. The result supports treating the agent's as a cost variable. The authors explicitly say the small task sample is insufficient to establish quality improvements, despite a promising pattern among stronger models.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
More from WRITER
Recommended reads
Comments

Checking sign-in…

Loading comments…