AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks
Source
arxiv.org
Author
Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu, Yi Xie, Nico Pelleriti, Shiyang Liu, Menghan Lu, Max Zimmer, Brando Miranda, Jiangchao Yao, Bo Liu, Sanmi Koyejo, Sebastian Pokutta, Bo Han
Date
Why it matters
Shows that wrapping a model in an agent harness does not reliably raise reasoning accuracy, so you can test tools, skills and memory separately instead of assuming they help.
Key takeaways · AI-distilled
The authors explain the inconsistency: agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → gains on questions the model could not solve alone can be outweighed by losses on questions it answered correctly without tools.
In component interventions under matched conditions, neither agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → guidance nor retained history reliably improved accuracy.
The authors warn tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → metrics must be read alongside correctness: low token entropy can accompany wrong answers, and less output per run can mean more output tokens per correct answer when accuracy falls.
The study covered three open models and three open harnesses on five answer-based benchmarks, with two models also run on two interactive benchmarks, keeping each harness's native control loop.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie
articleDeReAct: Decomposed Reasoning and Acting for Reliable AI AgentsAjay Vohra, Tao Chen, Neeti Narayan, Caron Zhang