Vibeleaderboard
← All Intel
Intel / article

How Public Evals Are Built Today and What Should Change

Source
eugeneyan.com
Date

Context

A public score measures the grader as well as the . The grader is the part of a benchmark that decides whether an agent's work passes, using code checks, a rubric scored by another model, or both. Eugene Yan went through twelve public agent benchmarks covering coding, science, office work and computer use, and in his survey he reports that in eight of them, audits or issue reports found the grader rejecting work that should have passed. In some it also accepted wrong or invented work. His argument is that benchmark authors have spent far more effort making tasks harder than making grading correct.

The most common grader error is failing correct work, usually because a check demands something the instructions never asked for. When Preference Model rebuilt 65 tasks from Terminal Bench 4.0, a benchmark of command-line work, it found 41 graded on an unstated requirement. Once those requirements were written into the prompts and shortcuts were closed, one model's score rose from 63% to 89%. Many benchmarks also use all-pass scoring, where a task counts only if every check passes, so a single bad check sinks an otherwise correct solution. In one Terminal Bench Science task, a run that classified 99 of 100 stars correctly scored zero.

The opposite error, passing wrong work, is harder to see because few people examine runs that pass. Most checks look for one expected point and ignore whether everything around it is correct. In AutomationBench, which tests agents running business workflows across simulated apps, a task that wants an email reporting how many audit items are ready and how many are not is graded only on whether the digits "3" and "4" appear, so an email that reverses the two numbers still passes. On Harvey's Legal Agent Benchmark, the judging model never sees the case documents. When Artificial Analysis checked deliverables against those documents, more than 60% of results that met every rubric item contained material hallucinations, such as a wrong party, amount or date. Earlier Intel covered that check: once hallucinations were counted, Muse Spark 1.3 fell from a 26.7% all-pass rate to 8.9%.

Yan also argues that the usual headline number, the average pass rate, hides how dependable an agent is. Two agents can both score 70%: one solves the same 70% of tasks every time, the other solves every task 70% of the time. A metric written pass^k counts a task only if all k attempts pass, and over five attempts it keeps the first agent at 70% while the second falls to 17%. The difference matters because agents often fail silently. In AutomationBench, models falsely declared success in 72% to 91% of the failures analyzed, so someone has to check every output. Scores also depend on the , the prompts, tools and limits wrapped around the model: on Terminal Bench 2.0, one model scored 57.8% in a neutral harness and 52.1% in its own provider's. Yan's conclusion is to use public scores to shortlist models and then test the shortlist inside the harness you actually run.

Key takeaways · AI-distilled
  • Eugene Yan found evidence in eight of twelve public he studied that graders failed valid work, usually through checks stricter than the prompt. Reviewers found 41 of 65 rebuilt Terminal Bench tasks graded on requirements the instructions never stated.
  • Average pass rate hides reliability. Yan's example: two agents both score 70%, but one always solves the same tasks while the other solves each task 70% of the time. Counting a task only if all five trials pass (pass^k) keeps the first at 70% and drops the second to 17%.
  • Yan argues cost per task misleads because agents can fail cheaply. In GDPval's analysis GPT-4o cost about a tenth of GPT-5 per task, yet once expert time to review and redo weak work was counted, GPT-4o was slower than an expert working alone while GPT-5 was faster.
  • A pass is not proof of correct work. On Harvey LAB, Artificial Analysis found material hallucinations in more than 60% of deliverables that passed every criterion, because rubric items check that an expected point is present and the judge never sees the source files.
  • Scores describe a model-harness pair, not a model. On Terminal Bench 2.0 one model scored 57.8% on the neutral Terminus 2 harness and 52.1% on its own provider's harness, so Yan recommends evaluating candidate models inside your own production harness.
Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source eugeneyan.com
Recommended reads
Comments

Checking sign-in…

Loading comments…