There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
Source
arxiv.org
Author
V. S. Raghu Parupudi
Date
Why it matters
A model's benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → score is a band, not a point — set partly by option order, prompt wording, and whether answers are read from text or logits. Leaderboard gaps narrower than that band are agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → artifacts.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleHarness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private SuiteMohsen Arjmandi
articleMeasuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?Jundong Hu, Shekar Ramachandran
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao