TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Source
arxiv.org
Author
Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
Date
Why it matters
Passing a fixed test suite can reflect adaptation to the evaluator rather than correct fixes. Per-trial evolved tests give a way to check whether benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → scores you rely on are inflated.
Key takeaways · AI-distilled
Rather than strengthening each task's tests once in advance, TestJack generates tests per trial that target prompt requirements the patch may violate, keeps only tests the ground-truth patch passes, and re-examines failures.
Every confirmed failure is backed by a replayable test. A lighter variant audits a random sample of trials in depth, then reuses those tests across all trials of the same task to cut evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → cost.
Across 6 frontier model backends and 5 benchmarks including DeepSWE and SWE Marathon, about 34.4% of trials judged correct violated task requirements, lowering the overall resolution rate from 50.6% to 33.2%.
The authors conclude that as models get better at optimizing against fixed evaluators, the evaluators themselves must become more adaptive.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleGameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State AssertionsXinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu
articleGroundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding AgentsJunyu Guo, Shangding Gu, Ming Jin, Javad Lavaei
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie