How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Source
huggingface.co
Date
Key takeaways · AI-distilled
AISI's release covers five benchmarks (HealthBench, FrontierMath, Humanity's Last Exam, SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Pro, and Terminal-Bench 2.0) across six models: Claude Opus 4, 4.5, and 4.6, plus GPT-5, 5.2, and 5.4.
Two additional cyber evaluations, Cyber CTFs and 'The Last Ones,' use a different, only partially overlapping set of models than the main five-benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → release.
The data accompanies AISI's paper on how inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → compute shapes evaluation, which found that when models get oracle correctness feedback after each attempt, they keep solving more Humanity's Last Exam tasks as token budgetA cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.Full definition → increases.
AISI publishes through EvalEval's Every Eval Ever schema and Evaluation Cards platform, providing verified configuration and transcript-level detail so others can reproduce or compare results across differing evaluation protocols.
Terms in this piece · Glossary
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
Why it matters
The UK AI Security Institute is now publishing evaluation results through EvalEval's shared schema, giving practitioners a structured, reproducible format for comparing model evaluations instead of one-off, hard-to-verify reports.
