What is the best harness for each model?
- Source
- runta.com
- Author
- Edward40
- Date

Context
A is the program that runs an AI model as an : it gives the model a task and tools such as a shell, executes what the model asks for and feeds back the results until the model says it is done. Codex CLI, OpenCode and Pi are examples. Runta's FrontierHarness Eval v1.1, built with the model host Baseten, asks whether the right harness depends on the model. It ran 22 harness configurations with five open models, GLM-5.3 and GLM-5.3 Flash from Z.ai, DeepSeek V4.1 Flash, Muse from Meta and Kimi-K3 from Moonshot AI, on 62 terminal tasks with four attempts each, 27,280 runs in all. For cost, the answer is yes. For results, paying more for a harness did not reliably help.
The model sets most of the bill. Runta's statistical split attributes 92.8% of the variation in cost per task to which model was used, because list prices differ by 20 times or more. But with the model held fixed, the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest, and shuffling trials between harnesses produced much smaller spreads, so Runta concludes the gap comes from the harnesses rather than rerun noise. The ranking also changes with the model: Codex CLI was the cheapest harness per task with GLM-5.3 and 16th of 22 with Kimi-K3. Each model got a different best-value harness, from Codex CLI with GLM-5.3 to Alma with Kimi-K3.
Spending more did not buy better results. Within each model, the rank correlation between a harness's cost per task and its pass rate ranged from −0.39 to +0.22, where a positive number would mean pricier harnesses pass more often, and with four of the five models the most expensive harness per pass actually passed less often than the cheapest. The cost driver was the number of steps, not the size of the prompt or the number of tools, because each step resends the whole conversation to the model. The smallest harness, mini-SWE-agent, sent the shortest first request but took 38 steps per pass and ranked 17th of 22 on cost per pass. Most money was wasted on failure: 64.5% of the cost went to runs that did not pass, and among the best-value harnesses, 68% to 82% of failed runs ended with a confident final answer rather than running out of time.
The findings carry limits Runta states itself. Only open models served by Baseten were tested, and its cache behavior and prices could reorder the cost rankings on another provider. Each pass rate has a 95% interval of about plus or minus 10 points, so small gaps are ties, and when Runta resampled the tasks, only Codex CLI with GLM-5.3 kept its best-value spot reliably, in 99.6% of draws; the other four winners held it in 25% to 53%. Runta's advice follows from that: when switching models, test candidate harnesses on your own tasks, and compare their cost before choosing.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…
