There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
Source
V. S. Raghu Parupudi
Author
V. S. Raghu Parupudi
Date
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
A model's benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → score is a band, not a point — set partly by option order, prompt wording, and whether answers are read from text or logits. Leaderboard gaps narrower than that band are agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → artifacts.
articleOne Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and ModelsSiqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, Martin Hirzel
postThe Harness Is Everything: What Cursor, Claude Code, and Perplexity Actually BuiltRohit