Vibeleaderboard
← Course overviewModule 08 · Make workflows dependable · Beta
Lesson 8.1

Module contents

Make workflows dependable

Lesson 1 of 5

What does finished look like before the work starts?

Module 3 ended with a delegation brief that named completion evidence for a single comparison. This module takes the same idea into work that recurs, takes longer and has more places to fail: a monthly report that reads several documents, does arithmetic and posts the result where other people see it. This lesson sets up that case and asks the question the rest of the module depends on. Before the workflow runs, what would a satisfactory result look like, and what could you observe afterwards that would show you got one?

Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.

What this lesson needs from earlier modules

Four ideas carry over. A fluent sentence claiming that something happened is a claim, not a receipt (module 1). A changes only when a writes it, and a request sent to a can succeed without any response arriving (module 2). A runs the a model asks for, records each result as an , and the is the evidence of what ran (module 3). A names and before the loop starts (module 3).

Module 4 explained how a harness loads an , keeps a , and carries work between contexts through a . Its names where needed information lives and what must survive a handoff. This module extends that plan to the conditions a result must satisfy and what happens after a failure.

Module 5 separated a tool’s stated behavior from its effective access, and explained why a missing response can make a retry unsafe. Module 6 provides a way to compare setups on representative work. Here the comparison becomes an ongoing check of a recurring workflow. The optional software-building module is not required: the worked examples use documents, invoices and a membership directory.

The recurring case: a monthly supplier invoice report

The garden's treasurer produces a report each month of what suppliers invoiced the garden that month. It records invoiced amounts, whether or not the garden has paid them yet; a report of payments made would be a different job with different inputs, and the treasurer's request has to say which one it wants. Everything about this case is authored for the module.

The treasurer keeps a short register of the suppliers expected to invoice in a given month, and for September it lists four: Northfence Ltd, Parkside Fencing BV, Seeds Co-op and Hedgerow Fencing. Three of them send invoice files that are saved in a folder. Hedgerow sends its invoice as an email attachment. The output is a text report with one row per supplier and a total in pounds, saved in the Reports folder and then posted to the committee's message board, where the committee reads it before its meeting. The treasurer wants to hand this job to an agent every month.

Here is the September folder as the example supplies it, with each invoice's contents beside its name so that you can check the module's arithmetic yourself.

File in Documents/Garden/Suppliers/Invoices/2026-09
northfence-invoice-0912.txt
Contents
Northfence Ltd. Invoice 4471. Fence panels and installation, north boundary. £2,150.00. Paid 12 Sep 2026.
File in Documents/Garden/Suppliers/Invoices/2026-09
parkside-invoice-0919.txt
Contents
Parkside Fencing BV. Invoice PF-1187. Gate hardware. €480.00. Due 30 Sep 2026.
File in Documents/Garden/Suppliers/Invoices/2026-09
seeds-coop-invoice-0903.txt
Contents
Seeds Co-op. Invoice S-2209. Autumn seed order. £64.50. Paid 3 Sep 2026.

The harness in this example has six tools: list_folder, read_file, write_file, fetch_email_attachment, which searches the garden's mailbox and returns a message and any attachment, post_report, which sends a file to the committee board and returns the board's confirmation, and list_board, which returns the board's current items with their ids, titles and contents. The board is a on another computer, so a post is a request over the network with everything module 2 said about responses. Reads run without asking; the write and the post ask first.

Notice what makes this case harder than the module 3 comparison. It recurs, so a mistake this month can recur next month. It combines reading, arithmetic and an action other people see. And one of its inputs arrives through a channel that can fail in a way that looks like an answer: a search that finds nothing is an observation, and what it means is a separate question.

A satisfactory outcome, written before the run

Module 3 asked, for a single task, how you will know it is right. For a recurring workflow the same question needs a fuller answer, because you will not be watching every run. The answer is a set of : statements, written before the workflow runs, of what the result must satisfy for you to accept it. Each criterion has to be something you could check against an actual result, which rules out criteria like "the report is good" and keeps ones like these, written for the September report:

Acceptance criteria: supplier invoice report, September 2026

1. The report has one row for each supplier on the September register. A row
   holds either the invoiced amount or a statement that the invoice was not
   obtained and why. An invoice found for a supplier not on the register gets
   a row marked "not on register" for the treasurer to resolve.
2. Every amount in the report matches the amount on the invoice it came from,
   in the currency the invoice states. Amounts not in pounds are shown in their
   own currency and are not added into the pounds total.
3. The total equals the sum of the pound amounts in the rows.
4. Every statement in the report about a budget, an approval or a decision names
   a document that was read during the run and that says so.
5. A missing attachment or unreadable file is reported as "not obtained" with
   the observation that shows it. It is never treated as zero, and it is never
   treated as evidence that the supplier did not invoice.
6. Exactly one copy of the report is on the committee board, and the copy there
   is identical to the file in the Reports folder.

Read the list for what it does. Criterion 1 bounds completeness to something the workflow can check: the register plus whatever records the run finds. A folder listing and a mailbox search cannot establish that no other invoice exists anywhere, and a criterion that pretended otherwise could never be met. Criterion 2 settles a meaning the treasurer's request left open, in the way module 3's brief settled that price meant installed price: a euro amount is not a pound amount, and the criterion says what to do with one. Criterion 4 is a rule about evidence, of the kind module 1 taught for a single reply, applied to a document the agent writes. Criterion 5 anticipates a failure and says what a correct report looks like when it happens. Criterion 6 is about the world outside the harness, and it will turn out to be the hardest to check.

OpenAI's guidance for people evaluating systems built on its models puts the same step first: define the objective and the success criteria before collecting cases or running anything. 1 That guidance is written for developers measuring a system across many runs; what it contributes here is the order of operations. Criteria come first, because a criterion written after you have seen a result tends to describe the result you saw. Criteria written first can still be missed, by the run and by the reader; they make a miss visible, which is what they are for.

Observable evidence for each criterion

A criterion is only useful if something observable can show whether it was met. So for each criterion, write down what you will look at, and notice that none of the answers is the agent's final message.

For criterion 1, the register, the folder listing and the email search result from the action record, compared with the rows in the report. For criterion 2, each invoice's text as returned by read_file, compared with the row it produced. For criterion 3, the arithmetic, redone. For criterion 4, the action record: if the report cites a document, a read of that document should appear as a call with an observation, and the observation should contain the claim. For criterion 5, the report itself, read against the fetch and read observations. For criterion 6, the board's contents, read with list_board or by a person opening it, compared with the saved file and with whatever is known about what the board held before the run.

The last one deserves a closer look, because it shows the difference between the final state of the work and the run's own story about it. The is what exists when the run ends: the file in the Reports folder, the items on the board, the mailbox as it was left. The run's completion message describes what the model believes it did. Anthropic's lesson for people running its coding agent unattended puts the rule in one sentence: do not start with the agent's summary of what it did, start with the actual changes. 2 That lesson is about reviewing code, where the changes are a difference between two versions of the files; module 7 covers that form of evidence for readers who take the software path. For a document workflow the actual changes are the saved report and the board's contents, and the same rule applies: read those, then read the summary and see whether it matches.

Module 2 gave a habit that makes criterion 6 easier to check: capture the before-state. If each run begins by recording what the board holds, a comparison afterwards shows what changed; receipts or activity logs help distinguish this run’s actions from changes made by other people. It is not the only evidence. A board that gives each item a stable id and keeps a log of who posted what and when can show the same thing after the fact, and a post tool that returns a receipt naming the item it created is evidence too. What matters is that some record independent of the model's message can establish what the run did to the destination, and that the workflow is designed so that such a record exists.

Why a confident completion message settles nothing

An agent saying it has finished does not show that the deliverable is correct. Module 1 explained the mechanism: the completion message is generated from the context, and the context contains the model's own plan, its calls and the observations that came back. A model can write "Done. Report written and posted" after a run in which the post received no response, because nothing in the process of generating that sentence checks the board. The sentence is consistent with the run and unsupported by it.

The sentence is also not worthless. It tells you what to check first. If the message says the report was posted, criterion 6 is the first thing to look at. If it says every supplier is in the report, count the rows against the register. A completion message is a set of claims to test against the final state and the record, in the order the claims are made, and that is exactly how the next lesson reads one.

Carry this forward

  • Write acceptance criteria before a workflow runs: statements about the result that can each be checked against something observable. A criterion written afterwards tends to describe what you saw; a criterion written first can still be missed, and makes the miss visible.
  • Bound completeness to something checkable, such as a register of expected suppliers plus the records the run finds. A missing input is reported as not obtained, never as zero and never as evidence that nothing was owed.
  • The final state is what exists when the run ends. The completion message is the model's account of it. Read the final state first, then test the message against it.
  • For an action on a destination outside the harness, make sure some record independent of the model's message can show what the run did there: a before-state captured at the start, stable item ids, the destination's own log, or a receipt from the operation.

Lesson check

The treasurer runs the September workflow, reads the agent's message "Done. All four suppliers are in the report and it is posted to the board," and then writes down the acceptance criteria. Which statement follows from this lesson?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

The criteria exist and the evidence for each is named. The next lesson takes the treasurer's first run, made before the criteria were written, and reads its record against them.

Sources · Reviewed 2026-09-29
  1. 1. OpenAI: Evaluation best practices
  2. 2. Anthropic Academy: Trust it: verifying unsupervised runs

Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.