Lesson 1 of 5
What must a setup do for this task?
Modules 3 to 5 took one harness apart: the loop, the layers, the information, the tools and their contracts. This module puts the parts back together and asks a practical question: for a particular piece of work, which combination fits? The answer starts with the work, not with a leaderboard or a product page. This lesson introduces the task the module compares setups against, writes down its requirements and constraints, and shows why those decide which measurements matter.
Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.
The task: reviewing three tenders
The garden committee has put its fencing job out to tender and received three submissions, each a document of ten to twenty pages: a price schedule, a method statement, insurance certificates, references and terms. The committee has a checklist of ten requirements every tender must meet, from the fence height and post spacing in the specification to public liability insurance above a stated amount and a start date within the season. The review is to read each tender against the checklist, list every discrepancy with a page reference, and total the priced items so that the three can be compared. Everything about the tenders, the checklist and the committee is authored for this module.
The organizer wants an agent to do the first pass, and a member to check it before the committee relies on it. That is a in module 3's sense: the inputs are three files and a checklist, the output is a discrepancy list with page references and three totals, the allowed operations are reading and writing one report, and the check is that a person can open each cited page and see the discrepancy, and add up each price schedule and get the report's total. What is new is that the organizer has to choose what to run it on, and there is more than one reasonable answer.
A in this module means the whole combination: the , the , the connections that give the harness its operations, and the path the tenders' contents take through all of it. Products bundle these in different ways. A hosted assistant may fix the model or offer a choice of several, and usually lets you add connections; a harness you run yourself may let you choose the model; a model you download fixes nothing about the harness. Comparing setups rather than products is what lets the comparison survive the next product release.
Four things a setup has to do for this task
Before looking at any candidate, write down what the task needs from a setup. Four things, for this review.
First, it has to produce a first-pass report a member can check and finish. The committee will judge the first pass by how many of the real discrepancies it catches, how many findings it reports that turn out not to be discrepancies, whether each finding carries a page reference a person can check, and whether the priced totals are right. The accepted report is the one after the member's review; the member is expected to catch what the first pass missed, so first-pass quality is measured by how much the member had to add and correct, not by whether the first pass was perfect. Note what quality is not, here: it is not fluency, and it is not the length of the report.
Second, it has to be affordable, and affordable has to be defined. The committee has set aside a small sum for provider charges, which this module writes as forty units; units are an invented currency for the lessons and are not any provider's price. Members' time is unpaid and finite. So the cost that matters is the total: any provider charges, including runs that fail and have to be repeated; the elapsed time a member waits, which may overlap other work; the time a member spends checking and correcting the report; and, for anything run on the committee's own machine, the power, hardware and maintenance that a zero provider charge hides. Lesson 4 gives this its name and a table.
Third, it has to be done by the next committee meeting, ten days away. That is a generous deadline; a setup that takes two hours is as acceptable as one that takes ten minutes. For a review needed during a meeting, the same setup would be judged differently.
Fourth, the committee's own policy for tender material says that nothing from a tender may be retained by anyone beyond the evaluation. The tenders contain contractors' insurance details and named staff. So the setup has to be one whose data handling the committee can state: where the documents go, who keeps them and for how long. This is a retention constraint. It is not, as written, a constraint on transfer: the policy does not say the documents may never be sent to a provider, only that no one may keep them afterwards. The two are different constraints, and lesson 2 shows why keeping them apart matters; lesson 5 changes one of them and watches what happens.
The fourth item is a : a requirement a setup must meet regardless of how well it performs otherwise. The deadline is another. A setup whose provider's documented terms say a year's retention, or that cannot finish within ten days, is out before quality is compared. Writing constraints down first is what stops the comparison being decided by whichever number is easiest to find.
Why the requirements come before the leaderboard
It is tempting to begin the other way round: find a published ranking of models, take the top one, pick the harness with the longest list of integrations, and call that the best setup. The module's later lessons say what a ranking measures. The point here is simpler: without the four requirements, nothing tells you which measurements are relevant. A ranking on coding tasks is indirect evidence about page-referenced discrepancy lists at best, not a measurement of them. An integration count does not tell you whether the one connection this task needs, a way to read three files, exists and what it may reach. A speed figure is irrelevant when the deadline is ten days and decisive when it is ten minutes.
The requirements also tell you what evidence to go and get. For this task the evidence that matters most is a trial on the actual kind of document: give a candidate setup one tender with known discrepancies and count what it finds and what it gets wrong. OpenAI's guidance on evaluation states the same principle for people building systems: compare candidates on tasks representative of the work, against criteria written down in advance. 1 The committee can do this with one tender and an afternoon.
The habit carries beyond this task. A setup is chosen for work; the work's requirements decide the dimensions of comparison; the dimensions decide which evidence is relevant; and the evidence, not the product page, decides. Lesson 5 writes the result down so that it can be revisited when the work, the models or the constraints change.
Carry this forward
- A setup is the model, the harness, its connections and the path the task's data takes, considered together. Compare setups for a task, not products in general.
- Write the task's requirements first: what counts as first-pass quality and what the member's review must add, what cost includes, how long the work may take, and where the data may go. Constraints such as the deadline and the retention policy are checked before quality is compared.
- A retention constraint and a transfer constraint are different requirements. The committee's policy as written limits who may keep the tenders, not whether they may be sent to a provider.
- The requirements decide which measurements are relevant and which evidence to seek. A trial on one representative document with known discrepancies is direct evidence about this task; a public ranking is indirect evidence at best.
Lesson check
Choose an answer for each question.
This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.
The requirements are written. Before candidates can be compared against them, the word setup has to be taken apart, because the questions about quality, cost and data land on different parts of it.
Sources · Reviewed 2026-09-29
Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.