Lesson 3 of 5
What does a benchmark score tell you?
Models arrive with numbers: a percentage on a named set of tasks, a position in a table. The numbers are real measurements, and the temptation is to read them as a verdict on the model. This lesson reads one as what it is: a count of something, under conditions, judged somehow. It then shows, with one vendor's experiment, that conditions the score does not mention can move it, and ends with how the committee can get evidence about its own task.
Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.
What a published score measures, and what it leaves out
A is a fixed set of tasks with a stated way of scoring them. A published score on one says: on these tasks, run this way, judged like this, the model achieved this. Each clause matters. Suppose an authored report says a model completed 71 of 100 tasks on a coding benchmark. The tasks were coding tasks, of a kind the benchmark's authors chose. They were run in some harness, with some time limit, on some machines. Completion was judged by some rule, perhaps a test that had to pass. The figure is evidence about all of that together. It is evidence about the committee's tender review only to the extent that reviewing tenders resembles those coding tasks in that harness under that judging, which makes it indirect evidence at best.
Read a score by asking four things. What were the tasks, and how like mine are they? What was the setup: harness, tools, time, machines? Who or what judged, and against what criterion? And what is not reported: how many attempts, what variance between runs, what the failures looked like? A published score that does not let you answer the four questions is a number without its units.
The judging clause has a modern complication. Many evaluations use a : a model reads the output and decides whether it meets the criterion, because a person cannot read ten thousand outputs. OpenAI's evaluation guidance, written for people building their own evaluations, says to check such a grader's verdicts against human labels before trusting it, and names two biases to watch for: favouring longer answers, and favouring one position when two answers are compared side by side. 1 Module 5's Supabase evaluation used a model grader, and the module said its limits belonged here. This is the limit: a score from a grader whose agreement with people has not been checked is a measurement of the grader as much as of the graded, and it should be read with that in mind rather than dismissed.
Conditions the score does not name
Even with tasks, setup and judging known, a score can move for reasons the report does not mention. One vendor's experiment makes this concrete. In February 2026 Gian Segato of Anthropic published an account of running the same model, the same harness and the same set of agentic coding tasks while changing only the computing resources the tasks ran on: how much memory and processing headroom each task had, and how strictly limits were enforced. The outcomes changed. Runs with more headroom failed less often for infrastructure reasons and completed more tasks; the report gives its figures and tests their significance for its own setup. 2 Those figures describe that vendor's tasks on that vendor's machines, and this course does not reproduce them as expectations for anyone else. The transferable finding is the shape of it: the environment an agent runs in is part of the task, and two runs of the same model on the same tasks can score differently because the machines differed.
For the committee this has two consequences. First, a published score was produced on machines and with limits the committee does not have, so even a perfectly matched benchmark would not predict the committee's result exactly. Second, a trial the committee runs itself, on the treasurer's laptop or through a provider, is a measurement under the committee's conditions, which is the measurement it wants. Module 4 made a related point about instruction changes: a comparison is an observation for those runs, and when results vary, repeating or varying the trial is what turns one observation into a pattern. The same applies to setups, and it applies to failures as well as to quality: which setup happens to time out or run out of memory in one trial is also an observation about that trial.
Getting evidence about your own task
The committee's evidence is a trial. Take one tender and plant discrepancies in it whose locations are written down: a fence height that does not match the specification on page 4, an insurance amount below the threshold on page 11, a start date outside the season on page 2, twelve in all across the checklist's ten items. Give each candidate setup the tender and the checklist with lesson 1's brief, allowing retries when a run fails, and score the completed report. Count what comes back: how many of the twelve were found; how many findings are not on the planted list, each checked by hand against the tender, since an unplanted finding may be a real discrepancy the organizer did not notice; whether the page references are right; whether the priced total matches the schedule when a member adds it; how many runs it took, how long the elapsed time was, what it cost; and how long a member needed to check and complete the report. That is an evaluation in the sense lesson 1 cited: representative work, criteria fixed in advance. It fits in an afternoon. Its limits are also plain: one document, one trial per setup, discrepancies the organizer planted rather than the ones real tenders contain. A second tender, or a second trial, tells you how much the first result moves.
The site's model picker and lab profiles are current comparison resources for the model part of a setup: what a model is documented to do, what its publisher and provider say about it, what has been measured and how. Use them after the requirements are written, to find candidates, and read any score there with the four questions of this lesson. Nothing in this course fixes which model is right for the tender review; a recommendation written today would need re-checking at each release.
The misconception this lesson replaces is that the highest score determines the best setup for every task. A score is evidence about its tasks and conditions. The best setup for the tender review is the one that meets the constraints and does best on the committee's trial, at a total cost the committee accepts, and the next lesson puts three candidates through that.
Carry this forward
- A benchmark score says: on these tasks, in this setup, judged this way, the model achieved this. Read it with four questions: what tasks, what setup, who judged, what is not reported.
- A model grader makes large evaluations possible and is itself generated text; one provider's guidance says to check it against human labels and watch for length and position bias.
- Conditions a report does not name can move a score. One vendor's experiment changed only computing resources and saw outcomes change on its tasks; the figures are its own, the shape transfers, and it applies to which setup fails as much as to which scores well.
- The committee's evidence is a trial on a representative document with planted discrepancies, with retries allowed and the completed report scored: found, unplanted findings checked by hand, references, totals, runs, time, cost and review effort. Site comparison pages help find candidates after the requirements are written.
Lesson check
Choose an answer for each question.
This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.
The committee ran its trial. The next lesson lays out what came back for three candidate setups and shows why the cost column has to hold more than the price of one model call.
Sources · Reviewed 2026-09-29
- 1. OpenAI: Evaluation best practices
- 2. Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals (Gian Segato, 2026-02-05)
- 12. VibeLeaderboard: Model picker (current model comparison)
- 13. VibeLeaderboard: Labs (lab profiles and current models)
Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.