Vibeleaderboard
← Course overviewModule 08 · Make workflows dependable · Beta
Lesson 8.3

Module contents

Make workflows dependable

Lesson 3 of 5

How would I know that a change helped?

The treasurer has a fix in mind for the currency failure: a line in the garden's project instructions saying that amounts not in pounds are shown separately and never added to the total. Next month's run goes well. Did the line help? Module 1 taught that one better retry does not identify its cause. This lesson turns that caution into a method: a set of representative cases, a baseline, a changed workflow, and a comparison that says what it can and cannot establish.

Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.

One good run is a single observation

Suppose the treasurer adds the currency line and runs October's report, and October's report is right. Several explanations compete, and the October run cannot separate them. The line may have helped. October's invoices may all have been in pounds, so the line was never exercised. Or generation varied: module 1 explained that the same context can produce different wording on different runs, and a model that wrote £480 in September might have written €480 in October with no change at all. And for any real workflow something else may have changed between the two runs, such as the provider updating the model behind the same name, or a supplier changing its invoice format.

This is the same reasoning module 1 applied to a reworded request and a better reply, applied now to a workflow you will run many times. A single successful run after a change is a single observation. It is consistent with the change helping and with the change doing nothing.

Representative cases

The way out is to stop relying on whatever this month happens to bring and to construct the months you need. A is an input, prepared in advance, that exercises something the workflow must handle, together with the result the criteria require for it. For the supplier report the treasurer can build a small folder of cases from past months and invented variations, each with a known correct report:

  • A typical month: three invoices, all in pounds, all as files. Correct report: three rows, the pound total.
  • A month with a euro invoice. Correct report: the euro row shown in euros, excluded from the total.
  • A month in which an emailed invoice has no attachment. Correct report: a row saying the invoice was not obtained, and no zero.
  • A month in which an invoice's text mentions an unrelated budget figure. Correct report: no budget statement, because no minutes were read.
  • A month with a supplier whose email subject says "statement" rather than "invoice". Correct report: that supplier's row present, which tests the search's scope.

OpenAI's evaluation guidance names these categories as typical cases, edge cases and adversarial cases, and warns against building a set of cases that does not resemble the situations the system actually meets. 1 The categories matter less than the reason for them: each case is aimed at a failure class from the previous lesson, so that a change meant for one class can be seen to affect that class. A case can exercise more than one class at once, since a euro invoice that also arrives by email tests interpretation and extraction together, and that is worth knowing when a case fails for a reason the change was not aimed at.

How many cases are enough is a question this course does not answer with a number, and the guidance does not either. Two considerations decide it. Each failure class you care about needs at least one case that exercises it, or the comparison is silent about that class. And because generation varies, a case run once tells you less than a case run several times, so the number of runs matters as much as the number of cases. A small set run once gives limited evidence about how often a fix will hold up when generation varies. The honest statement of what a small set establishes is in the last section.

A baseline and a changed workflow, run on the same cases

A runs the unchanged workflow and the changed workflow on the same representative cases, under the same conditions, and compares the results against the criteria case by case. The unchanged workflow is the baseline. To attribute a difference to one component, that component has to be the only thing that differs: the same model, the same harness, the same tools, the same cases, and the currency line present in one and absent in the other. If the model also changed between the two, any difference has two candidate causes and the comparison cannot choose between them. Comparing two whole setups is also a legitimate comparison; it answers a different question, whether the new setup does better on these cases, and says nothing about which part made the difference.

Here is the treasurer's comparison as an authored illustration. Each case was run three times in each condition; the table records how many of the three runs met every criterion. These numbers are invented to show the shape of a result, and they are not measurements of any product.

Case
Typical month
Baseline (no currency line)
3 of 3
With currency line
3 of 3
Case
Euro invoice
Baseline (no currency line)
0 of 3
With currency line
3 of 3
Case
Missing attachment
Baseline (no currency line)
0 of 3
With currency line
0 of 3
Case
Unrelated budget figure
Baseline (no currency line)
1 of 3
With currency line
1 of 3
Case
Subject says "statement"
Baseline (no currency line)
0 of 3
With currency line
0 of 3

Read the table as evidence about the change. The euro row moved from none to all, across three runs, with nothing else different: in these authored runs the line was associated with the improvement it was written for, and the case that improved is the one it addresses. The other rows did not move, which fits, since the line says nothing about attachments, citations or email subjects. Had the missing-attachment row also improved, that would call for investigation rather than celebration: the line gives no mechanism for it, and until the runs are read, variation, an unnoticed second change and an indirect effect of the line are all open.

The budget row is the awkward one: one of three in both conditions. That is consistent with the workflow sometimes inventing a budget statement and sometimes not, regardless of the line, and three runs cannot say more. It says where the next fix is needed, and it says that a single October run showing no budget statement would have been consistent with either condition.

What the comparison establishes, and what it does not

State the result in the strongest form the evidence supports and no stronger. The comparison supports: with this model, this harness and these cases, adding the currency line was associated with the euro case going from failing in three of three runs to passing in three of three. It does not support: that the line fixes currency handling in general, that it will hold when the model behind the same name changes, that it helps with any other failure class, or that a change on one case run three times is beyond what variation could produce. A larger number of runs would narrow the last doubt; it would not touch the others.

Two practitioner reports show the same discipline from different directions, and each is scoped to what it observed. A talk given at an AI Engineer event in July 2026 by a speaker recorded as Yogi, whose full identity the site's Intel entry does not verify, describes skills, meaning saved procedures a harness loads for a job, as versioned contracts with a model: an unchanged skill can start producing different behavior after the model is upgraded, and the speaker's remedy is a set of checks that runs after every upgrade. 6 The site's entry is a summary of a transcript, and the transcript may contain errors. The claim that transfers is that a comparison made against one model version is evidence about that version, and the module 1 principle behind it is that the same context can produce different output when the learned values change. The treasurer's currency line is a project instruction whose effect also needs checking after a model change, and its comparison should be run again when the model changes.

The second report comes from Anthropic. In March 2026 Prithvi Rajasekaran, of the company's labs team, described a harness for building applications over long runs and a method for testing it: remove one component at a time and re-run, on the grounds that every component encodes an assumption about what the model cannot do on its own, and a newer model may not need it. The account reports that one component, which broke work into sprints, could be removed for a newer model without losing performance. 7 That is a vendor's experiment on its own harness with its own models, and the specific finding does not transfer. The method does, and it is the currency table run in reverse: change one thing, hold the rest, compare on the same cases. It also cuts the other way for the treasurer. The line added for this model may be unnecessary, or harmful, for the next one, and the way to find out is the same comparison.

An unchanged command does not guarantee identical output. The command is one input. The model's learned values, the provider's serving of it, the tools' behavior and the random element in selection are others, and a workflow that ran correctly last month has not been shown to run correctly this month until this month's evidence is read.

Carry this forward

  • One successful run after a change is one observation, consistent with the change helping, with the change never being exercised, with variation, and with something else having changed.
  • Representative cases are prepared inputs with known correct results, each aimed at a failure class, so that a change meant for one class can be seen to affect that class. A case can exercise more than one class.
  • A baseline comparison runs the unchanged and changed workflows on the same cases under the same conditions. With one difference it can associate that component with a change in results; with several it can only compare whole setups. State its result for those cases, that model and that harness, and no wider.
  • A comparison is evidence about the model version it was run against. Practitioners report unchanged procedures behaving differently after upgrades, and re-running the comparison is how to find out.

Lesson check

In the authored comparison table, the euro case went from 0 of 3 to 3 of 3 after the currency line was added, and nothing else was changed. Which conclusion does the lesson support?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

A comparison needs someone or something to judge each result against the criteria. The next lesson asks who can do that, and what each kind of checker can and cannot establish.

Sources · Reviewed 2026-09-29
  1. 1. OpenAI: Evaluation best practices
  2. 6. VibeLeaderboard Intel: Skills as the interface to agentic products (AI Engineer talk, July 2026)
  3. 7. Prithvi Rajasekaran, Anthropic (24 March 2026): Harness design for long-running application development

Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.