Jan Kulveit, Gavin Leech, Tom\'a\v{s} Gaven\v{c}iak, Raymond Douglas
Date
Context
An AI system's ability to complete work independently is a different evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → target from how well people work with it. This position paper argues that benchmarks should focus more on human-AI teams. The authors contend that emphasizing autonomous performance steers development toward replacing people, while collaborative evaluation would reward systems that complement human capabilities.
The proposal changes what counts as progress: the outcome of the partnership becomes the object of evaluation. Better societal outcomes are the authors' rationale for this shift, rather than a measured result reported in the abstract.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.