Xue Li, Hongqian Xia, Daniel Gomm, A\'ecio Santos, Cornelius Wolff, Madelon Hulsebos
Date
Why it matters
Accuracy-only evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → hide data leakage and invalid validation in data science agents. The released test suite lets you check for these specific failures before trusting AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → output.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleWhen Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic SystemsWenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
articleAgentic Data Engineering for LLMs: System Design and a Controlled Empirical StudyRuomeng Ding, Qianli Shen, Zhaoyang Han, Meixin Chen, Daoyuan Chen, Yaliang Li