Vibeleaderboard
← Course overviewModule 08 · Make workflows dependable · Beta
Lesson 8.5

Module contents

Make workflows dependable

Lesson 5 of 5

How does the work survive failure and interruption?

The first September run posted the report twice because a response went missing and the model tried again. This lesson is about that moment and the others like it: a run that stops halfway, a host that restarts, a retry that costs money. It adds four things to the supplier workflow, saved progress, safe retries, budgets and stopping conditions, ends with a transfer task in a different setting, and closes with the record that carries this module into the next.

Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.

A missing response is not a failure; it is an unknown

Line 18 of the September record said "no response after 30 seconds." Module 2 explained what that observation means for a request to a server: the request may never have arrived, or the server may have acted and the reply was lost on the way back. The observation cannot tell you which. Line 19 treated it as a failure and sent again, and the board's log shows that the first post had succeeded.

So the first rule for an operation with an , module 3's term for a consequence outside the harness, is that an unknown outcome is not a reason to repeat the operation. It is a reason to find out. Finding out means inspecting the destination: read the board with list_board and look for an item that matches the file. In the September case that inspection would have found item 7730, and the workflow would have stopped there with one copy.

The inspection has a limit of its own. Finding exactly one item for this report, with the intended contents and evidence tying it to this request, resolves whether this post happened. Conflicting or duplicate items need investigation. If it is not there, the question is not settled, because a post the board received a moment ago may still be processing or not yet visible in a listing, and a missing item read immediately after a timeout does not prove that the first request failed. What the workflow does then depends on what the board guarantees. If the board offers a way to ask about the status of a request, or honors an identifier as the next paragraph describes, the workflow can use it. A status query helps only when its result resolves the request: a pending or unknown status remains uncertain. An identifier must be reused for the same intended action, not recycled for a different report revision. If it offers neither, an unresolved absence is a stop condition: record what was observed and hand the decision to the treasurer, who can wait and look again, rather than post a second time on the strength of a listing that may be stale.

Amazon's engineering library describes the underlying problem for services that create resources on request: a caller that times out cannot tell whether the resource was created, and a naive retry can create two. Its remedy is a contract between caller and service. The caller supplies a unique identifier with each request; the service records the identifier together with the change it makes, as a single indivisible step, so that a repeated request carrying the same identifier is recognized and answered with the original result instead of performed again. The article also notes the conditions: the service has to implement this, it decides how long it remembers identifiers, and a repeat with the same identifier but different contents should be refused. 9 This course calls such an identifier an . Module 5 names the property it gives the service, an operation whose repetition has the same effect as doing it once, an ; the identifier is one way a service provides it.

The condition is the part that matters for the treasurer. If the board's post_report tool accepts an identifier unique to this particular publication, such as garden-september-report-revision-1 and the board honors it, then repeating the same request within that contract’s scope and retention period need not create a second post, and the second post at line 19 would have returned item 7730 again. If the board has no such contract, then putting an identifier in the request, or in the model's instructions, changes nothing about what the board does. Whether a retry is safe is a fact about the receiving service, found in its documentation or by testing, and not a property a workflow can give itself. Where the service offers no contract, the workflow falls back to the first rule: inspect, act only on a positive finding, and stop on an unresolved one.

Saved progress, and what a progress note cannot promise

The September run was short enough to finish in one . A report that reads fifty invoices, or a task that waits on a slow service, may not be. Module 4 asked what survives between sessions and answered with the , whatever carries work from one context to the next; module 2 established what happens to state held only in a process's memory when the process ends. Saved progress is the answer: a record, written to storage as the work proceeds, of which steps are complete and what evidence shows it, so that a later run can continue rather than start over. This course calls such a record a , Here checkpoint means workflow state, not the learning exercises elsewhere in the course. The storage behind that record determines whether it survives a restart.

Anthropic's November 2025 account of running its coding agent across many sessions describes the plain form: a progress file the agent updates as it works, a list of features with each marked done or not, and version-control snapshots with descriptive messages, so that each new session begins by reading the progress file and the recent history, then running the existing checks before starting anything new. The account reports that without such a record the agent tended to declare the project finished early and to leave work in a broken, undocumented state. 8 That is one team's experience with one coding harness and the models of that time. What transfers is that the progress record lives in files, that it names what is done and what evidence shows it, and that the next session reads it before acting.

The word checkpoint also appears in software frameworks for building agents, and there it does not always mean something saved. LangChain's documentation for its LangGraph framework distinguishes checkpoints, which snapshot a run's state at each step, from a store that keeps data across runs, and it says of its in-memory checkpoint options that they keep checkpoints in the computer's working memory and lose them all when the process restarts; the options that survive a restart write to a database. 10 Nothing in this course requires that framework, which was read and not installed. The point is the module 2 point: calling something a checkpoint does not say where it is held, and a checkpoint held in memory is gone with the process.

Even a checkpoint written to storage has a limit that the September run makes visible. Suppose the workflow wrote a progress note before posting, saying "report posted: pending," and the process stopped after the post was sent and before the note was updated. A later run reads "pending" and posts again. The note and the board disagree, because the note records the workflow's intention and the board records what happened, and a failure between the two leaves them out of step. Infrastructure exists that takes this seriously: Temporal, a system for running long workflows, keeps a persisted history of every event and, after a failure, rebuilds a workflow's position by replaying that history rather than trusting any single note. 11 Even there, the replay restores the workflow's own position, not the world's: a step that calls an outside service can still have run without its result being recorded, and such systems expect those steps to be made safe to repeat, or reconciled against the destination, by the person who builds them. Temporal explicitly documents this retry case and recommends idempotent activities. 12 For the treasurer's workflow the practical rule is the one from the previous section: a progress note tells the next run where to look, and for any step with an external effect the next run looks at the destination before it acts.

Budgets and stopping conditions

A workflow that can retry and can resume can also run away: retrying a fetch forty times, resuming a run that keeps failing at the same step, or spending a month's allowance of paid model calls on one report. A is a limit, set before the run, on what one run may consume: how many retries per operation, how many steps in total, how much time, how much money where the model or a tool is paid for by use. A budget is enforced by the harness where the harness supports it, and where it does not, it is an instruction, with everything module 3 said about instructions.

Budgets work together with stop conditions, which module 3 introduced as situations named in advance in which the agent stops and hands the decision back. The September run needed three that it did not have. If an expected input is not obtained after the allowed retry, stop and report which one, rather than deciding that the supplier did not invoice. If an amount is in a currency the report does not handle, show it separately, as the criteria now say, and if that cannot be done, stop. If an operation with an external effect returns no response, inspect the destination, continue only on a positive finding, and stop on anything unresolved. Each stop condition converts a situation the model would otherwise resolve by itself into one a person resolves, or a rule resolves.

Here is the supplier workflow's recovery plan, written for the four September failures and the interruption case. Where a line below sets a specific limit, it takes precedence over the general budget at the end.

Recovery plan: supplier invoice report

Missing or unusable input (attachment absent, file unreadable or garbled)
  evidence:   the fetch or read observation, quoted in the report
  action:     one retry, then a row saying "not obtained" with the reason
  stop:       if the register still has a supplier without a usable invoice
              after the retry, write the report with the "not obtained" row
              and flag it to the treasurer before posting

Unhandled currency
  evidence:   the invoice text beside the row
  action:     show the amount in its own currency, excluded from the total
  stop:       if any row's currency does not match its invoice, do not post

Unsupported statement
  evidence:   a read_file call with a non-error observation for every cited document
  action:     remove any statement without one
  stop:       do not post a report that cites a document not read in the run

External effect with unknown outcome (post: no response)
  evidence:   list_board, compared with the before-state, plus the board's
              log and any receipt the post returned
  action:     if exactly one item matches this publication and its intended
              contents, record the receipt and finish. Investigate duplicates.
              Otherwise query the original request's status if supported.
              Retry only after a definitive failed/cancelled outcome, or under
              the service's documented idempotency contract with the same key
              and payload within its retention period.
  stop:       do not repeat a request that could still be pending without that
              protection. Report the unresolved outcome and time to the treasurer.

Interruption (process or host stops mid-run)
  evidence:   the progress note and the folder and board as they now are
  action:     re-read inputs; skip the write if the file exists and matches
              the intended contents; inspect the board before any post
  stop:       if the note and board have an unexplained disagreement, do not
              post; report both. A confirmed receipt can settle a pending note.

General budget for one run, where no line above is tighter:
  2 retries per operation, 40 tool calls, 15 minutes, and the spending
  limit the treasurer sets for paid calls.

The plan is written in words, and each line has to be matched by something in the harness where possible: a retry limit the harness enforces, a post tool that accepts an identifier, an ask rule before any post. Where the harness cannot match a line, the line stays an instruction, and the treasurer knows which lines those are. That is module 3's rule for a brief, applied to a plan.

Transfer checkpoint: a membership directory update

Try the module's distinctions on a workflow that is not the supplier report. Everything here is authored.

A choir keeps a members directory as a file. Each month an agent updates it from a form export, a spreadsheet of new sign-ups downloaded from the choir's website, then emails each new member a welcome message. The harness has read_spreadsheet, read_file, write_file and send_email; reads run freely, the write and each email ask first. This month's run has this record, abbreviated:

 1  [model: call]    read_spreadsheet   Choir/signups-2026-09.csv
 2  [observation]    4 rows: name, email, joined
 3  [model: call]    read_file          Choir/members.txt
 4  [observation]    82 rows
 5  [model]          Row 3 has joined = 03/09/2026; I will read it as 9 March.
 6  [model: call]    write_file         Choir/members.txt   (86 rows)
 7  [harness]        Asking before a file write.   [person] approved
 8  [observation]    wrote file: Choir/members.txt
 9  [model: call]    send_email         to: 4 addresses   subject: Welcome to the choir
10  [harness]        Asking before sending 4 emails.   [person] approved
11  [observation]    error: no response after 30 seconds
12  [model]          The emails may not have been sent. Stopping and reporting.

The final state: members.txt has 86 rows, one of which has the same name as an existing member and a different email address; the email service's sent folder shows four messages with the subject "Welcome to the choir", addressed to the four new addresses, timed at line 9. The export's documentation, which the run did not read, is not available to you; assume the date format is unknown.

Before reading the guide, write your own answers to five questions. First, classify what happened at line 5, at line 6, and at line 11, using the failure classes from this module. Second, for the date and the same-name row, what acceptance criterion and what checker would have caught each? Third, which of the run's operations can safely be repeated on a re-run, which cannot, and what must be inspected before deciding? Fourth, was line 12 the right response, and what should the next run do first? Fifth, outline the record the choir should keep of this run: cases, observed errors, evidence, recovery, limits.

Self-review guide. A complete answer classifies line 5 as an interpretation failure of a particular kind: the observation was correct, the export did not say whether its dates are day-first or month-first, and the model chose a reading without a criterion to guide it. While the format is unconfirmed the choice is unsupported, not wrong; once the export's format is known, 03/09 is a valid date in whichever reading applies. The criterion that fixes this names the format the export uses, and the checker rejects an ambiguous reading only while the format is unresolved.

It classifies the same-name row at line 6 as a missing criterion rather than an error the model made: nothing said what to do with a sign-up whose name matches an existing member, and a name is not a unique identifier, so the right behaviour is to flag the match for a person to resolve, neither merging the rows nor rejecting the sign-up automatically.

It classifies line 11 as an external effect with an unknown outcome, and it reads the sent folder carefully: four messages to the four new addresses at the time of line 9 is strong evidence that the service accepted and sent them, which is what a sent folder records; it is not evidence that any recipient received them, and four matching subjects alone would be weaker than the matching recipients and timing that the example supplies.

For repetition, it says the reads can be repeated; the write cannot be repeated on the strength of a saved before-state alone, because a re-run has to check the file as it now is, compare it with the intended contents, and look for any change another person made since, or a second overwrite may destroy their work; and send_email must not be repeated, because the sent folder resolves the unknown in this case. It says line 12 was right: the model stopped at an unknown outcome instead of resending, and the next run's first step is to read the sent folder and the progress note before touching anything.

For the record it lists the cases the choir needs, including an ambiguous date and a returning member; the two problems and the near-miss; the evidence for each, meaning the export rows, the file as it was and as it is, and the sent folder; the recovery plan with an inspect-before-resend rule and a stop on unresolved absence; the limit that the sent folder is the email service's record of sending and not of delivery; and a proposed improvement, the date criterion, with a baseline comparison on the cases before it is trusted. An answer that names the classes without saying which line's evidence shows each is not yet complete.

Keeping the record, and what comes next

Everything this module produced for the supplier report can be kept together: the representative cases and their correct results, the failures observed in real runs with the evidence for each, the acceptance criteria and which checker tests each one, the recovery plan with its budgets and stop conditions, the limits of what has been established, and any proposed improvement together with the baseline comparison that would test it. This course calls that collection a for convenience; the name matters less than the contents, and any form that keeps these records together, with a date and a model version on each, does the job. It extends module 4's , which said where each thing a task needs is kept and what must survive a handoff, with what must be true of the result and what happens after a failure. The date and version matter because a comparison and a judge are both evidence about the setup they were run against.

The record is not proof that the workflow works. It lets you say, for a given month, what was checked, what passed, and what remains untested, and it is what changes when a practitioner's tip is tried: the tip goes in as a proposed improvement, the comparison is run, and the result is recorded beside it with its conditions. Two tips from this module's sources, the progress file from November 2025 and the removable component from March 2026, are already in that form, each attached to the failure it addresses and dated. They were not replaced by each other; a newer model may not need what an older one needed, and the record keeps both with their dates so that the question can be asked again.

The record also answers a question this module has not asked. Once one workflow's inputs, checks and effects are written down, it becomes possible to see which of its parts depend on which. The invoice reads do not depend on each other. The total depends on all of them. The post depends on the file. That structure decides whether the work could be split across several agents or sessions, and whether splitting it would help, and it is the next module's subject. Work whose parts cannot be checked is hard to divide, because the seams are where checks have to go.

Carry this forward

  • A missing response leaves an operation’s outcome unknown. Inspect the destination and any request status. Retry only when the outcome or the service’s documented duplicate-prevention behavior makes that safe; an unresolved absence is a reason to stop.
  • Saved progress lives in files the next run reads before acting. A checkpoint held in memory is gone with the process, and even a saved progress note, or a replayed history, records the workflow's position rather than the destination's, so the destination is checked for any step with an external effect.
  • A run budget limits retries, steps, time and spending, set before the run and enforced by the harness where it can be. Stop conditions turn situations the model would resolve alone into ones a person or a rule resolves, and each needs a real stopping action.
  • Keep the cases, observed failures with evidence, criteria and their checkers, the recovery plan, the limits and any proposed improvement with its comparison together, each dated and tied to a model version. That record is what makes the workflow's structure visible enough to divide.

Lesson check

A post to the committee board returns no response. The board's documentation says nothing about recognizing repeated requests or reporting a request's status. The treasurer's instructions to the agent include the sentence "always include the report id so that posts are never duplicated." What follows from this lesson?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

You have reached the end of module 8. The module quiz below checks how these ideas fit together before module 9 asks when one workflow should become several.

Module 8 quiz: put the ideas together

1. Which of these is an acceptance criterion in this module's sense?
2. After a run, the treasurer has the agent's completion message, the action record, the saved report and the committee board. In what order does this module say to read them, and why?
3. A report states "the committee approved £3,000 for the fence (minutes, 14 August)". The action record, which the example supplies as complete, contains no read of any minutes, and the harness loads nothing into the context beyond the brief and the observations. Which statement follows?
4. After the first September run, the treasurer adds one sentence to the request: "If a post gets no response, check the board before posting again, and never post twice." What does this module say that sentence can and cannot do?
5. The treasurer adds a currency line in September and, between September and October, the provider releases a new version of the model behind the same name. October's report is right. What can be concluded?
6. An automated check is meant to confirm that every document the report cites appears as a read_file call with a non-error observation in the record. It passes. What has been established?
7. A framework's documentation says its default checkpoint option keeps checkpoints in the process's working memory. The workflow using it stops halfway when the host restarts. What does this module predict about resuming?
8. What does the record this module asks the treasurer to keep, the cases, failures, criteria, checkers, recovery plan and comparisons with their dates, establish about the supplier workflow?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

Sources · Reviewed 2026-09-29
  1. 9. Amazon Builders' Library: Making retries safe with idempotent APIs
  2. 8. Anthropic engineering (26 November 2025): Effective harnesses for long-running agents
  3. 10. LangChain documentation: LangGraph persistence
  4. 11. Temporal documentation: Workflow execution
  5. 12. Temporal: Activity definitions and idempotency

Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.