Lesson 4 of 5
Make a check fail for the right reason, then pass
A program that prints a plausible table is not evidence that it does what the specification says; the flawed version in this lesson printed one. This lesson writes checks from lesson 1's behavior examples, runs them against a version of the program with a one-line flaw, watches the right checks fail, traces the failure back along the data path, fixes it and watches them pass. It then asks the harder question: what does a passing set of checks establish, and what does it not. The checks, the flawed version and a `float` version are authored for the course; every result quoted was produced by running them offline on 2026-09-29 on the two fixture files printed in lesson 2.
Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.
A check states behavior, not implementation
A is a small program that gives the real program an input from the specification and compares the result with the expected output. Here is the file written for the quote comparison. It imports the program by name so that the same checks can be pointed at a different version, and it reads both fixture files from the folder it is run in.
"""Behavior checks for compare_quotes. Run from the folder holding the two
fixture files: python3 check_compare.py [module_name]"""
import importlib
import sys
from decimal import Decimal
name = sys.argv[1] if len(sys.argv) > 1 else "compare_quotes"
cq = importlib.import_module(name)
results, problems = cq.compare(cq.read_quotes("quotes.csv"), "40")
by_name = {r["supplier"]: r for r in results}
checks = {
"installation added when not included":
by_name["Hedgerow Fencing, Ltd"]["installed_price"] == Decimal("2280"),
"installation not added when included":
by_name["Northfence Ltd"]["installed_price"] == Decimal("2150"),
"lower installed price comes first":
[r["supplier"] for r in results] == ["Northfence Ltd", "Hedgerow Fencing, Ltd"],
"different length is reported, not compared":
"Parkside Fencing" not in by_name and any(p.startswith("row 4:") and "38 m" in p for p in problems),
"quoted supplier name with a comma is read as one field":
"Hedgerow Fencing, Ltd" in by_name,
}
messy, messy_problems = cq.compare(cq.read_quotes("quotes-messy.csv"), "40")
checks["unreadable price is reported by row"] = any(p.startswith("row 2:") and "2,150" in p for p in messy_problems)
checks["missing installation price is reported by row"] = any(p.startswith("row 3:") and "installation_price" in p for p in messy_problems)
checks["installation_included other than yes or no is reported by row"] = any(p.startswith("row 5:") and "yes or no" in p for p in messy_problems)
checks["comparable rows are still compared when others fail"] = [r["supplier"] for r in messy] == ["Hedgerow Fencing, Ltd"]
failed = 0
for label, ok in checks.items():
print(("PASS " if ok else "FAIL ") + label)
failed += 0 if ok else 1
print(f"{len(checks) - failed} of {len(checks)} checks passed for {name}")
sys.exit(1 if failed else 0)
Each check is named after the behavior it tests, in the words of lesson 1: installation is added when not included, a different length is reported rather than compared, a quoted name with a comma is read as one field, comparable rows are still compared when others fail. All nine go through read_quotes on a real file, so the quoting behavior is tested through the CSV reader and not by handing the program a ready-made dictionary.
Two other kinds of check are worth recognizing. One asserts only that installed_price returns a value: it passes for any implementation that returns something, including one that returns the wrong number, so it confirms that the function ran and nothing else. The other computes the expected value with the same expression the program uses, price plus installation price, and compares. That check would catch the flaw below, because the program's wrong answer and the check's expected sum differ; its weakness is coupling. Whoever writes the expected value by copying the program's expression will copy the program's mistake if the expression itself is wrong, and an agent that writes both the code and the checks in one pass is in exactly that position. Writing the expected number down from the specification, 2280 for Hedgerow, keeps the check's knowledge separate from the program's.
Red before green
Simon Willison's guide to working with coding agents describes the discipline as writing the tests first, confirming that they fail, then iterating on the implementation until they pass, and names the risk of skipping the failing run: a test that already passes may never have exercised the new code (red/green TDD, checked 2026-09-29). That is his assessment from his own use; the mechanism is what transfers. A is one you have seen fail before you saw it pass, and the point of seeing it fail is to know the check can tell the difference.
Seeing it fail is necessary but not sufficient. The failure has to be for the intended reason. Here is what happened when the check file was copied to a folder without the fixture files and run there:
Traceback (most recent call last):
File "check_compare.py", line 10, in <module>
results, problems = cq.compare(cq.read_quotes("quotes.csv"), "40")
File "compare_quotes.py", line 15, in read_quotes
with open(path, newline="", encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'quotes.csv'
No check ran at all: the file is read on line 10, before the loop that prints results, so the exception ended the run with a traceback and exit status 1. That tells you the fixture was missing, whether because the checks were run from the wrong folder or because the file was never copied, and it tells you nothing about the program. When a check first fails, read its message and confirm that it names the behavior you meant to test.
For this lesson a flawed version of the program was authored deliberately: identical to the delivered one except that installed_price returns the quoted price whether or not installation is included. Run on the three-row file, it printed a tidy table with Hedgerow first at 1980 and Northfence second at 2150, exit status 1 for the Parkside length, and nothing else. A reader who did not know Hedgerow's installation was extra would accept it. The checks, pointed at the flawed version, printed this:
FAIL installation added when not included
PASS installation not added when included
FAIL lower installed price comes first
PASS different length is reported, not compared
PASS quoted supplier name with a comma is read as one field
PASS unreadable price is reported by row
PASS missing installation price is reported by row
PASS installation_included other than yes or no is reported by row
PASS comparable rows are still compared when others fail
7 of 9 checks passed for compare_quotes_flawed
Two checks failed, and each names a behavior from the specification. Seven passed, which is also information: the flaw did not touch reading, reporting or the handling of bad rows. The failure is for the intended reason, and the message says which requirement is broken.
From the failing check to the line
is following a failure back along the data path to the place where the data first goes wrong. The first failing check says installation was not added for Hedgerow. Lesson 2 located the installed-price rule in one function, so that is where to look. Comparing the two versions of the program line by line, the only difference is this:
< return money(row["price"]) + money(row["installation_price"])
---
> return money(row["price"])
The second failure, the wrong ordering, needs no separate fix: with Hedgerow's total wrong at 1980, it sorted ahead of Northfence, and the correct total of 2280 puts it second. One cause, two symptoms, which is common and worth expecting; fix the cause and rerun everything rather than patching each symptom.
Restoring the line and running the checks again printed nine passes. That is the green half of the pattern, and it means something specific: the nine behaviors named in the file hold for the delivered program on those two fixture files. The whole cycle, from the flawed table to the passing run, went through the specification twice, once to write the checks and once to read the failure, which is why the specification had to exist first.
When an agent makes a change, the same cycle applies to the change. Ask for the failing run before the fix, not just the passing run after it. If the agent's report says only that all checks pass, you do not yet know whether any check exercised the change.
What a passing run does not establish
A passing run establishes the checks that are in it, on the inputs they use. It says nothing about behaviors that have no check. The program's handling of a name written across two lines was observed in lesson 2 and is still unchecked, because the specification never decided it. Nine green lines can sit beside it.
The float question from lesson 3 shows the same limit from the other side. An authored version of the program using float throughout its numeric conversions and a corresponding finite-number check was run against the same nine checks, and all nine passed: every fixture amount is a whole number of pounds, on which the two types agree. A tenth check was then written with a price of 2149.99 and an installation price of 0.02, and run against both versions. The Decimal version returned 2150.01. The float version returned 2150.0099999999998, which fails a check that the total is exactly the text 2150.01 and passes a check that the total rounds to 2150.01. Neither outcome was knowable from the nine green lines; it became known by writing a check for the behavior in question and observing the result, and which of the two criteria counts is the committee's decision, not the type's.
That is the answer to the belief that a passing test suite proves every requirement. A suite establishes the cases it contains. When reviewing one, pick a requirement, find the check that would fail if it were broken, and if there is none, the requirement is unobserved however green the run looks.
The agent that wrote a change can review it usefully, by rereading the diff against the specification, and should. What it cannot supply is a judgment uncorrelated with its own: the same assumptions that produced the change will tend to approve it. The harness write-up cited in lesson 3 reports that a generating agent grading its own work tended to praise it confidently where a human observer found it mediocre, which is why that team built a separate evaluator (harness design, March 2026). means assessment from something that did not produce the change: a person reading the diff against the specification, a check the person wrote from the behavior examples, or a second agent given the specification and the diff without the first agent's reasoning. None of these guarantees an error-free result either; they reduce the chance that one mistaken assumption passes unchallenged. The failing run before the fix is itself a form of independent evidence, because it came from the checks rather than from the author.
Carry this forward
- A behavior check gives the program an input from the specification and compares the result with an expected output written down from the specification, not computed with the program's own expression.
- See a check fail before you see it pass, and confirm the failure is for the intended reason; a missing fixture ends the run with a traceback before any check reports.
- Debug by following the failure back along the data path; one cause often produces several symptoms, so fix the cause and rerun everything.
- A passing run establishes the cases it contains; the
floatversion passed all nine whole-pound checks and diverged only on a pence check that was written and run. - An author can review its own change usefully, but independent review, from a person, a check or a second reader that did not produce the change, is what challenges the author's shared assumptions.
Lesson check
Choose an answer for each question.
This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.
Checks say whether the program does what was asked. The last lesson asks what else must be true before the committee relies on it: how it treats input it should not trust, what it depends on, what it must never contain, and how to get back if a release turns out to be wrong.
Sources · Reviewed 2026-09-29
- 3. Simon Willison: Red/green TDD, from Agentic Engineering Patterns
- 4. Prithvi Rajasekaran (Anthropic engineering): Harness design for long-running application development (March 2026)
Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.