Meta-Harness: End-to-End Optimization of Model Harnesses
- Source
- yoonholee.com
- Author
- Yoonho Lee
- Date
Context
Meta- searches for better software around a model by examining previous attempts. Its project report describes an that can inspect candidate code, scores and execution logs, propose a revision, and test it. Detailed logs give the optimizer evidence about where an attempt failed.
In a math experiment, the system optimized how examples were retrieved and presented before a model solved a problem. On 200 held-out competition-level problems across five models unseen during search, the authors report average accuracy of 38.8%, versus 34.1% without retrieval and 37.5% with keyword-based retrieval. This supports a specific finding: an automatically discovered retrieval procedure transferred to additional models in that .
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
“Meta-Harness takes a different approach: it gives the proposer a filesystem containing the full source code, scores, and execution traces of every prior candidate.”
Yoonho Lee
“In practice, this means up to 10M tokens of diagnostic context per step, vs. at most 26K for all prior methods we surveyed.”
Yoonho Lee
“On Claude Opus 4.6 , Meta-Harness achieves 76.4% pass rate, surpassing Terminus-KIRA (74.7%) and ranking #2 among all Opus 4.6 agents.”
Yoonho Lee
“A single discovered retrieval harness improves accuracy by +4.7 points on average (34.1% → 38.8%) across five held-out models.”
Yoonho Lee
Checking sign-in…
Loading comments…