As benchmarks saturate, scores stop separating models. This work tests whether solved-task transcripts still carry capability signal, which bears on how evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → can stay useful.
Key takeaways · AI-distilled
Of the protocols tested, pairwise judging of transcripts worked best, cutting prediction error 5.3 times compared with a baseline that used only the saturated benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → scores.
The capability signal is spread across transcript sections but concentrated in particular channels, notably the reasoning and the tool calls.
The predictions do not always hold under distribution shift: the authors report the protocols underestimated the first reasoning models released in early 2025.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie
articleTestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator EvolutionShuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song