Lesson 2 of 5
What did the run actually do?
The treasurer's first September run was made before the criteria in the previous lesson existed; the request was the plain sentence "Produce the September supplier invoice report and post it to the committee board." Its message says the report is written and posted. This lesson reads that run's action record and final state against the criteria, retrospectively, finds four failures, and shows that they are four different kinds of failure with different fixes. The point is not that this run was bad. It is that "the agent got it wrong" names nothing you can act on, while "the attachment was never retrieved" does.
Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.
The authored record of the first September run
Here is the run, in the trace format module 3 introduced: one line per event, each labelled with what produced it. Everything is authored; no software produced this record, and the example supplies it as complete, with nothing omitted. The record is what the treasurer has to work from; the criteria came later, and reading an old run against new criteria is a legitimate use of them, as long as nobody concludes that the absence of criteria caused each failure. A run with the criteria in front of it can miss them too.
Before the run, the treasurer listed the board: three older items, 7688, 7702 and 7715.
1 [model] I will read this month's invoices, fetch Hedgerow's from email,
write the report and post it.
2 [model: call] list_folder Documents/Garden/Suppliers/Invoices/2026-09
3 [observation] northfence-invoice-0912.txt parkside-invoice-0919.txt seeds-coop-invoice-0903.txt
4 [model: call] read_file .../northfence-invoice-0912.txt
5 [observation] "Northfence Ltd. Invoice 4471. Fence panels and installation, north
boundary. £2,150.00. Paid 12 Sep 2026."
6 [model: call] read_file .../parkside-invoice-0919.txt
7 [observation] "Parkside Fencing BV. Invoice PF-1187. Gate hardware. €480.00.
Due 30 Sep 2026."
8 [model: call] read_file .../seeds-coop-invoice-0903.txt
9 [observation] "Seeds Co-op. Invoice S-2209. Autumn seed order. £64.50. Paid 3 Sep 2026."
10 [model: call] fetch_email_attachment from: accounts@hedgerow-fencing.example
subject contains: invoice
11 [observation] 1 message (18 Sep 2026, "September invoice"). Body: "Please find our
invoice attached." Attachments: none.
12 [model] Hedgerow has not invoiced the garden this month.
13 [model: call] write_file Documents/Garden/Reports/supplier-invoices-2026-09.txt
contents:
Supplier invoices, September 2026
Northfence Ltd £2,150.00 (invoice 4471)
Parkside Fencing £480.00 (invoice PF-1187)
Seeds Co-op £64.50 (invoice S-2209)
Total £2,694.50
Note: the committee minutes of 14 August approved £3,000 for
the fence, so the fence work is within budget.
14 [harness] Asking before a file write. [person] approved
15 [observation] wrote file: Documents/Garden/Reports/supplier-invoices-2026-09.txt
16 [model: call] post_report board: committee file: .../supplier-invoices-2026-09.txt
17 [harness] Asking before posting. [person] approved
18 [observation] error: no response after 30 seconds
19 [model: call] post_report board: committee file: .../supplier-invoices-2026-09.txt
20 [harness] Asking before posting. [person] approved
21 [observation] posted. board item 7731
22 [model] Done. The September supplier invoice report is written and posted to
the committee board. Total £2,694.50.
And here is the final state, as the example supplies it. The Reports folder contains the file at line 15. The board now shows the three older items and two new ones, 7730 and 7731, both titled "Supplier invoices, September 2026" with identical contents. The board's own activity log, which the example also supplies, records both as posted from the garden's account, 7730 at the time of line 16 and 7731 at the time of line 19. The mailbox is unchanged.
Four failures, read against the criteria
Take the criteria in order and look for the evidence the previous lesson named.
Criterion 1, one row per supplier on the register. The register names four suppliers; the report has three rows. The listing at line 3 shows three files, and the search at line 11 found a Hedgerow message that says an invoice is attached and returned no attachment. Line 12 is where this went wrong: the model turned "attachments: none" into "has not invoiced," a claim the observation does not support. The message body says the opposite. Whether Hedgerow invoiced is unknown from this run; what is known is that the report has no Hedgerow row and no statement that the invoice was not obtained.
Criterion 2, amounts in the currency the invoice states. Line 7 says €480.00. The report row says £480.00. The record shows a correct input and a wrong output; it does not show what happened in between, and the course does not claim to see that. The total at line 13 adds the figure in as pounds: 2,150.00 plus 480.00 plus 64.50 is 2,694.50, which is the total the report shows. The arithmetic is right and the sum is wrong, because one of its terms should not have been in it. Without the Parkside row the pound total is 2,150.00 plus 64.50, which is 2,214.50, with €480.00 shown separately.
Criterion 4, every budget statement names a document that was read and says so. The report cites committee minutes of 14 August. No line in the record reads any minutes, and in this harness the model's context is the brief plus the observations, so there is nothing else the claim could have come from. Within this authored, complete record the claim is unsupported. It may even be true, if such minutes exist and say so, but nothing in this run looked, so the report has no basis for it: an unsupported addition of the kind module 1 called a . The boundary of that conclusion matters. In a setup where the harness also loads project notes or an into the context, as module 4 describes, the claim might have a source there, and the check has to cover those too; and an absence from a partial log proves nothing, which is why the example says the record is complete.
Criterion 6, exactly one copy on the board. There are two. Line 18 reported no response after thirty seconds; line 19 sent the same post again; the board shows 7730 and 7731, and its log attributes both to the garden's account at the times of the two posts. So in this example the first post succeeded and its confirmation was lost, which is precisely the case module 2 described when it said a missing response leaves the server's action unknown. Retrying created a second copy. Without the log, two matching items would be consistent with that and with someone else posting a copy; the example supplies the log so that the conclusion is earned.
Criteria 3 and 5 are worth checking too. The total does equal the sum of the rows, so criterion 3 passes on its own terms, which shows why criterion 2 was needed as well. Criterion 5 fails, because the missing invoice was neither reported as not obtained nor given a row.
The completion message at line 22 is true about the file, misleading about the board, and silent about Hedgerow, the currency and the minutes. Read first, it would have sounded fine.
Four different kinds of failure
The four failures are not four instances of one problem. Each is shown by different evidence and calls for a different fix. Naming the kind is the first step, and this course calls the kind a . The classes are defined by what the record shows, not by anything about the model's internal computation, which module 1 said is not readable.
The missing attachment is an : the information the workflow needed did not enter the context correctly, because the operation meant to bring it in returned without it. The evidence is the observation at line 11. The same class covers an input that arrives malformed, such as a scanned invoice whose text comes back garbled: present in the context, but not usable. The wrong currency is an : the observation at line 7 is correct and the output built from it is wrong. The evidence is the comparison between line 7 and the row at line 13. These two look alike from the outside, because both produce a wrong report, and they are fixed in different places. An extraction failure is fixed by getting a usable input, or by reporting that it could not be got. An interpretation failure is fixed by a criterion or instruction about how to treat that kind of input, and by a check that compares output with input afterwards.
Mistral's cookbook for reading documents with a text model shows the two stages as separate steps in a built system: the model asks a document-reading tool to run optical character recognition, which turns an image of text into text, and then answers from the returned text. 3 If the recognition step returns garbled text, the answer fails at extraction; if the returned text is right and the answer misreads it, the failure is interpretation. The cookbook is a demonstration for one provider's tools and says nothing about how reliable either stage is; the two-stage shape is what transfers.
The fabricated minutes are a third class, an unsupported claim: the report asserts something no observation or supplied context supports. The evidence is an absence in a complete record. Module 1 taught this check for a chat reply; here it applies to a document the agent wrote, and criterion 4 is the rule that makes it checkable.
The duplicate post is a fourth class, a repeated : an operation with a consequence outside the harness was performed twice because its first result was unknown. The evidence is the board and its log. The fix is in what the workflow does after an unknown outcome, which the last lesson of this module builds.
One more class from module 2 belongs on the list even though this run does not show it: a scope failure, where an operation completes and answers a narrower question than the one asked. Module 2's account of Simon Willison's January 2026 report is the standing example, in which a search of one site could not establish that a draft was unpublished everywhere. 4 In this workflow, a search of the mailbox for "invoice" in the subject is a scope failure waiting to happen the month Hedgerow titles its email "September statement."
Diagnose before you change the prompt
The tempting response to a run like this is a longer request: "Read all the invoices including email attachments, use the right currencies, only cite documents you have read, and do not post twice." That sentence addresses all four failures, and it is worth being exact about what it can do. Module 3's three layers give the distinction. An instruction shapes what the model proposes, and that is not nothing: an instruction can direct the model to retry a fetch, to search the mailbox a second way, to stop when an attachment is missing, or to inspect the board before posting again, and a capable model may do some of those unprompted. What an instruction cannot do is enforce any of it. It does not make an absent attachment present, it does not stop a retry the harness allows, and it cannot give the board a way to recognize a repeated post. Wording guides choices; it does not guarantee a boundary.
So the question for each failure is which layer the fix belongs in, and whether an instruction is enough there. Andrew Ng, a computer scientist who teaches machine learning and writes a newsletter for DeepLearning.AI, argued in October 2025 for looking before changing: build the workflow, run it, examine a handful of outputs by hand to see where it stumbles, and only then decide what to measure and what to change. 5 That is a practitioner's argument about how to spend effort, and it matches what this lesson did: read the record, name each failure's class, and let the class choose the fix.
The fixes the classes choose are different in kind. For extraction: a retry and then a stop condition when an expected input is not obtained, so the run reports rather than invents; an instruction can state this, and a harness retry limit enforces the ceiling. For interpretation: the currency criterion, which is an instruction, plus a check that compares each row with its invoice, which is not. For the unsupported claim: a rule about citing, plus a check of the record for a read behind each citation. For the repeated effect: a rule to inspect before repeating, which an instruction can state, and duplicate protection at the board, which only the board can provide. Every fix has an instruction in it and none is only an instruction. The next three lessons take up how to compare a proposed fix with the run it is meant to improve, who can perform each check, and how to make the retry safe.
Carry this forward
- Compare the action record and final state with the criteria, then test the completion message against them. The first September message correctly reported a saved file and a board post, but omitted the duplicate post and the report’s errors.
- An extraction failure means a usable input did not reach the context; an interpretation failure means the observation was correct and the output built from it was wrong. Both are read from the record, and they are fixed in different places.
- An unsupported claim is shown by an absence in a complete record: no read, and no supplied context, behind the citation. A repeated external effect is shown by the destination and its own evidence, such as a log or a before-state, not by two matching items alone.
- Name the observed failure before choosing a fix. Instructions can guide searches, retries and stops; tool behavior, access controls and independent checks can support or enforce the required behavior.
Lesson check
Choose an answer for each question.
This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.
Four failures have names and each points at a fix. Before applying one, the next lesson asks how you would know that a fix helped, rather than that the next run happened to go better.
Sources · Reviewed 2026-09-29
- 3. Mistral AI cookbook: Document comprehension with any model via tool usage and OCR
- 4. Simon Willison (12 January 2026): First impressions of Claude Cowork
- 5. Andrew Ng, DeepLearning.AI (15 October 2025): Improve agentic performance with evals and error analysis, part 1
Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.