How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Source
huggingface.co
Date
Context
A benchmark score is easier to interpret when its testing conditions travel with it. The UK AI Security Institute and EvalEval collaboration puts publicly reported results into Evaluation Cards, using a common reporting structure to retain methods, configuration and context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → alongside the scores.
That matters when apparently comparable results used different amounts of model computation or different evaluation procedures. The release accompanies research on those differences. Making the setup visible gives researchers a way to examine whether two scores answer the same question before treating one as evidence of a better model.
Key takeaways · AI-distilled
AISI's release covers five benchmarks (HealthBench, FrontierMath, Humanity's Last Exam, SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Pro, and Terminal-Bench 2.0) across six models: Claude Opus 4, 4.5, and 4.6, plus GPT-5, 5.2, and 5.4.
Two additional cyber evaluations, Cyber CTFs and 'The Last Ones,' use a different, only partially overlapping set of models than the main five-benchmark release.
The data accompanies AISI's paper on how inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → compute shapes evaluation, which found that when models get oracle correctness feedback after each attempt, they keep solving more Humanity's Last Exam tasks as token budgetA cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.Full definition → increases.
AISI publishes through EvalEval's Every Eval Ever schema and Evaluation Cards platform, providing verified configuration and transcript-level detail so others can reproduce or compare results across differing evaluation protocols.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
