Harness evals: cost differs more than pass rate across agent harnesses

Garry Tan, who runs Y Combinator, has published harness-evals, an open repository that asks whether the harness around an AI model changes how well it does. A harness is the program wrapped around the model: its system prompt, its tools and its memory. Claude Code and Codex are harnesses; so are OpenClaw, Hermes Agent, Pi and QM, the others tested here. Tan's runs keep the model (Claude Opus 5.5 or GPT-6.1 Sol) and the tasks fixed, swap only the harness, and record how often each solves a task, what each solved task costs and how long a try takes. His headline finding, which he shared on X, is that harnesses differed more on cost than on how often they solved tasks.

Pass rates were mostly hard to separate. On 52 public Terminal-Bench 2.1 coding tasks, each harness was compared only with the model maker's own harness, Claude Code for Opus and Codex for Sol, using tests fixed before the run. Five of the six passed within 15 points of it, a result that means at most 15 percentage points worse, not equal, and for three of them the whole 95% range of the gap sat below zero. OpenClaw on Opus fell behind Claude Code, 76.0% against 88.5%, though the README calls that result fragile: it becomes inconclusive if tries the model provider refused are dropped, or if tries hit by a GitHub outage during grading are handled under one alternative rule alone. On 20 multi-user tasks, where the agent acts as a simulated company's chat bot answering several people, no tested pair differed, although gaps smaller than 7.7 to 14.2 points could go undetected. Cost varied more. On those multi-user tasks, OpenClaw on Opus cost $0.44 per solved task against $0.19 to $0.24 for the other Opus harnesses. The README traces about 81% of OpenClaw's extra cost over Claude Code on Opus to cache writes, the charge for storing prompt text so later requests can reuse it, because OpenClaw re-sends a much bigger system prompt and bigger tool results on each request. The cost differences carry no statistical test.

The runs also tested where instructions should go. A rule to check gbrain, Tan's shared memory for agents, before answering was followed in 30 of 30 turns on several harnesses when it sat in the system prompt or a per-turn hook, a setting that adds text to every message (26 of 30 through Claude Code's hook). A skill nobody mentioned could go unread: Claude Code opened one 0 times in 30 turns. Tan's own gstack, a pack of workflow skills for coding agents, did not change what Opus did. Told to run gstack's /investigate, /qa, /review or /ship, Opus read the skill text every time and followed none of its steps in 46 sessions, across three small checks run outside the formal runs, at about twice the cost per try. Because the tasks were easy for Opus, the README says this shows the steps were not followed, not whether following them would help. Tan has argued for putting reusable procedures in skill files in Thin Harness, Fat Skills, so this result tests his own approach.

The README presents these as early results. The tasks are public and were visible while the harness was being built, so they count as leads rather than confirmation on hidden tasks, and they hold only for the pinned harness versions. On the multi-user tasks each harness ran with its shipped settings, so those rows mix harness design with different thinking and effort levels. Tan also discloses that gstack and gbrain are his projects, that QM is a Y Combinator project, that Y Combinator invests in Hermes and that the simulated company comes from his gbrain-evals. Every try's records are in the repository, and its reproduction steps rebuild the tables without calling a model or needing an API key.