Vibeleaderboard
← Course overviewModule 08 · Make workflows dependable · Beta
Lesson 8.4

Module contents

Make workflows dependable

Lesson 4 of 5

Who can check the work, and what can each establish?

Every criterion needs a checker. For the treasurer's five cases run three times each, comparing the original and changed setup produces thirty reports to judge, and next month there will be more. This lesson looks at the three kinds of checker available, a person, a program and a model, and asks of each the question this course keeps asking: what does its verdict actually establish? The answer is different for each, and the differences decide which criteria to hand to which.

Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.

Human review takes attention

A person reading the September report against its invoices can catch all four failures in the previous lessons, because a person can ask what the report means: is €480 the same as £480, does "attachments: none" mean no invoice, did anyone read those minutes, and how many copies are on the board? Human review is the checker that can judge whether a criterion was met in the sense the criterion intended, rather than in the sense a rule can express. It is also the checker that can miss all four, by reading the total, finding it adds up, and moving on. A person's reading is evidence in proportion to how carefully and how expertly it was done, and a treasurer who knows the suppliers reads differently from a volunteer who does not.

Human review takes time and attention. Assign people to questions where interpretation matters, such as whether the cited minutes support the report’s budget claim. They may also catch mistakes outside the planned criteria. Periodically compare automated or model verdicts with a person’s reading, then resolve disagreements using the source evidence.

An automated check establishes what it actually exercised

An is a program that reads the output, the record or the destination and reports whether a stated condition holds. It needs no model. A program can apply the same explicit rule across many cases without attention fading. Its behavior still depends on its inputs, implementation and environment. Its limit has two parts: it establishes only the condition it tests, and only if it was written correctly, pointed at the right artifact and given the right expected values. A check can be buggy, can run on last month's file, or can carry a wrong answer key, and a pass from such a check establishes nothing. What a pass establishes is what a correctly implemented check actually exercised, so the check itself is something to test, on a case known to fail.

The currency failure shows why the condition has to be written with the failure in front of you. A first attempt at a check might be: every amount in the total has a pound sign in its row, and the total equals their sum. Run it on the first September report. The Parkside row reads £480.00, with a pound sign. The total, 2,694.50, is the sum of the three rows. The check passes, and the report is wrong. The relabelling that caused the failure also satisfies the condition, because the condition only looks at the report. The check that catches it compares each row with its source: for every row, the currency symbol and amount in the row match the currency symbol and amount in the invoice text returned by read_file for that supplier. Run on the same report, that check fails on Parkside, because line 7 says €480.00 and the row says £480.00.

The same care applies to scope. A check that every file in the folder has a row misses Hedgerow, whose invoice never was a file, and would wrongly demand a row for a stray file that is not an invoice. The check that matches criterion 1 starts from the register: for every supplier on the September register, the report has a row with an amount or a row marked not obtained; and for every invoice the run read that names a supplier off the register, the report has a row marked not on register. For criterion 4: every document the report cites appears as the input of a read_file call with a non-error observation in the record, and, since the harness in this example loads nothing else into the context, that read is the only place the claim could have come from. None of these checks would catch a report that cites the right minutes and misreads what they say. A check is only as good as the condition it tests, and writing the condition is the work.

For coding work, Anthropic recommends having the harness run tests rather than accepting an agent’s statement that they passed. Its unattended-runs lesson also separates an action-safety reviewer from checks of the resulting work. 2 The supplier workflow needs that distinction too: permission to post says nothing about whether the invoice amounts are correct.

A model judge is a second generated opinion

A is a model asked to read an output and say whether it meets a criterion. Module 6 called this a model grader; the names refer to the same role. It is attractive because it can read for meaning, as a person can, and it scales, as a program does. Its verdict is generated text, subject to everything module 1 said about generated text, and how much it is worth depends on how the judge was set up and on what it is asked to judge.

OpenAI’s evaluation guidance recommends checking a model judge against human judgments on shared cases, and warns about preferences for response order and length. 1 A padded report could receive a better verdict than a concise, correct one. Anthropic reports that its coding agent often praised its own output; a separately calibrated evaluator improved its review process but still missed defects. That is evidence from one team’s applications. 7

A fresh reviewer session can inspect the output and criteria without inheriting the author’s conversation. 2 It can still share the author’s errors. Self-review can find real problems, but neither self-review nor another model’s approval proves that none remain. Compare verdicts with human readings of the same cases and resolve disagreements against the invoices and record. Human reviewers can be wrong too. The question is which judgments the evidence supports.

Assigning criteria to checkers

Put the three together and each of the September criteria has a natural checker. Criteria 1, 2 and 3 are conditions a program can test against the register, the invoices and the report, once the conditions are written to catch the failures the previous lessons found. Criterion 6 is a program's job too, and it is not a count. Two items on the board is a symptom; the criterion is that exactly one item matches this report, so the check reads the board with list_board, finds every item whose contents match the saved file, and confirms there is one, using the board's ids and log where it can, because a count alone cannot tell a duplicate from someone else's post. Criterion 4 needs a person or a validated model judge, because it is about whether a document says what the report claims. Criterion 5 is partly mechanical, since a program can check that every register supplier has some row, and partly a judgment about whether the row's explanation is honest.

One rule of thumb follows from the differences. Use automated checks where an explicit rule is practical and valuable, and test each check on a case known to fail before trusting its passes. Consider a model judge for criteria that require interpretation rather than a simple mechanical comparison, and measure it against a person's readings first. Reserve human review for the criteria where meaning decides, and for the periodic check of the other two checkers. A workflow with only a completion message has no checker at all; a workflow with all three may still have criteria none of them tests, and the next run may fail there.

Carry this forward

  • Human review can judge whether a criterion was met in the sense intended, and can also miss what a rule would catch. It is expensive and attention falls with volume, so assign it to the criteria where meaning decides.
  • An automated check establishes what a correctly implemented check actually exercised, and nothing beyond that condition. A check can be wrong, so test it on a case known to fail. A check that looks only at the output can pass the currency relabelling; the one that catches it compares each row with its source invoice.
  • A model judge reads for meaning and scales, and its verdict is imperfect evidence. Measure it against a person's readings, resolve disagreements against the source evidence, expect verbosity bias, and treat a model's review of its own output as useful but not independent.
  • A reviewer that checks each action for danger does not check whether the work is correct. Safety checking and correctness checking are separate jobs.

Lesson check

The treasurer asks a fresh model session to read each of thirty reports and say whether every amount is in the right currency. It passes all thirty. What does that establish?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

Three of the four September failures now have checks. The fourth, the duplicate post, is not a checking problem. It is a question of what the workflow does when it does not know what happened, and the last lesson answers it.

Sources · Reviewed 2026-09-29
  1. 1. OpenAI: Evaluation best practices
  2. 2. Anthropic Academy: Trust it: verifying unsupervised runs
  3. 7. Prithvi Rajasekaran, Anthropic (24 March 2026): Harness design for long-running application development

Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.