Vibeleaderboard
← Course overviewModule 06 · Choose a setup that fits · Beta
Lesson 6.4

Module contents

Choose a setup that fits

Lesson 4 of 5

Which setup did the work, at what cost?

Three candidate setups each ran one trial on the planted tender. This lesson lays out what happened in a table whose every figure is invented for teaching, and reads the table column by column. The reading turns on one idea: the cost of a setup is everything the task consumed to reach a report the member accepted, and the price of the final model call is only part of that.

Numbered links identify sources. Underlined terms offer a definition without leaving the lesson.

Three candidate setups

The organizer chose three arrangements that differ in the parts lesson 2 named. All three are fictional, and none corresponds to a real product, price or plan.

Setup A: a large hosted model, reached through a full-featured harness the organizer already uses, with a connector to the folder holding the tenders. The tenders' contents go to the model's provider. The organizer read and recorded the provider's documented API terms on the day: not used for training by default, retained for a stated number of days for abuse monitoring, not sent onward. Whether that retention period satisfies the committee's policy that nothing is kept beyond the evaluation is an open point the organizer has to put to the committee, so A is a conditional candidate.

Setup B: a smaller, cheaper hosted model from the same provider, in the same harness with the same connector. Same data path and the same open point as A; the only difference is the model.

Setup C: an open-weights model whose license the organizer checked permits this use, run locally on the treasurer's laptop through a minimal harness with a file tool, a shell tool and a web-search tool that the organizer had not switched off. The model's computation stays on the laptop. What else the harness does with data, its logs, any telemetry, the laptop's backups, and what the shell tool could reach, the organizer had not yet checked, so C is a conditional candidate too.

Each setup ran one trial: the planted tender and the checklist, with lesson 1's brief, retried when a run failed to complete, and scored on the completed report. A run that completed is one that produced a report, not one that got everything right.

Reading the table

Here is what came back. Every number is teaching data, invented so that the columns can be read; units are an invented currency, not any provider's price; and the arithmetic within each row is consistent.

Measure (teaching data)
Planted discrepancies caught
A: large hosted
11 of 12
B: small hosted
8 of 12
C: local open-weights
9 of 12
Measure (teaching data)
Findings not on the planted list, checked by hand
A: large hosted
1, of which 1 real
B: small hosted
3, of which 0 real
C: local open-weights
1, of which 0 real
Measure (teaching data)
Page references wrong
A: large hosted
0
B: small hosted
2
C: local open-weights
1
Measure (teaching data)
Priced total matches the schedule
A: large hosted
yes
B: small hosted
no
C: local open-weights
yes
Measure (teaching data)
Runs: completed / failed
A: large hosted
1 / 1
B: small hosted
1 / 0
C: local open-weights
1 / 2
Measure (teaching data)
Provider charges (units)
A: large hosted
8.50
B: small hosted
1.10
C: local open-weights
0.00
Measure (teaching data)
Of which failed runs (units)
A: large hosted
2.10
B: small hosted
0.00
C: local open-weights
0.00
Measure (teaching data)
Elapsed waiting, all runs (minutes)
A: large hosted
23
B: small hosted
6
C: local open-weights
105
Measure (teaching data)
Member's review time (minutes)
A: large hosted
25
B: small hosted
40
C: local open-weights
35
Measure (teaching data)
Where the tender's contents went
A: large hosted
provider
B: small hosted
provider
C: local open-weights
laptop; one web search left

Read the quality rows first, because the task is the quality. A caught 11 of 12 planted discrepancies and reported one finding that was not on the list; the member checked it against the tender and it was a real discrepancy the organizer had not planted, which is why unplanted findings are checked rather than counted as errors. A's page references were right and its total matched the schedule when the member added it. The member's 25 minutes went on opening twelve pages, confirming the findings, adding the total, and finding the one planted miss by reading the tender against the checklist with the planted list in hand. B caught 8, reported three unplanted findings that all turned out not to be discrepancies, got two page numbers wrong, and its total did not match the schedule: one priced line was left out. The member spent 40 minutes checking, discarding, correcting the references and the total, and locating four misses. C caught 9 with one unplanted finding that was not a discrepancy and one wrong reference, and its total matched; 35 minutes.

The totals row is there because lesson 1's task asked for three priced totals, and a report whose total is wrong cannot be accepted until a member has re-added the schedule. In this trial that check was done for all three; B's report needed correcting before it could be accepted at all.

Now the runs row. A's first run failed when the connector timed out fetching the tender; the harness reported the failure, and the second run completed. That failed run cost 2.10 units and nine minutes of the 23, and both are in A's totals, because the committee paid for them. C failed twice: the laptop ran out of memory on the first two attempts, each after about thirty-five minutes, and the third completed after the organizer closed everything else. B ran once. Which setup failed and how often is one observation from one trial each; a second trial could see A run clean and B time out, and the lesson does not read the failure column as a property of the models.

The row is elapsed time across all runs, from asking to having a completed report. It is waiting, not labor: during C's 105 minutes the treasurer did other things and came back. C's elapsed time is well within lesson 1's ten-day deadline; for a review needed in a meeting it would rule C out by itself.

The last row is lesson 2's data path, observed rather than promised. A and B sent the tender's contents to the provider, whose documented terms the organizer recorded and whose fit with the committee's policy is the open point. C kept the model's computation on the laptop, but the harness's web-search tool sent one query, the name of an insurer from page 11, to a search service. That single query is not automatically harmless: it tells the search service that someone on that machine was looking at that insurer, and the committee's policy was about tender material. Nothing else was observed to leave, and the harness's logs, telemetry, backups and shell access had not been checked, so “nothing else left” is what the trial observed and not what the setup guarantees.

Total work cost is not the price of the final call

If cost meant the provider's charge for the completed run, the ranking would be C at nothing, B at 1.10, A at 6.40. That ranking compares a part of each setup. is everything the task consumed to reach an accepted report: every run including the failed ones, the elapsed waiting, the member's review and correction time, whatever the committee's own machine cost to run, and the risk from anything the member did not catch. Counted that way for this trial, A consumed 8.50 units, 23 minutes of elapsed time and 25 minutes of review. B consumed 1.10 units, 6 minutes and 40 minutes of review. C consumed no provider charges, 105 minutes of elapsed time, 35 minutes of review, and the laptop's power and the organizer's set-up time, which the table does not price. Whether B is cheaper than A depends on what the committee makes of 15 extra review minutes against 7.40 fewer units, and that is a judgment the committee has to make explicitly rather than let the invoice make for it. The table does not assign a monetary value to time or hardware, so it cannot establish what fraction of the whole cost each provider charge represents.

Misses need care in the accounting. In the trial the member found every planted miss, because the planted list was in hand; those misses cost review minutes and are already counted there. In a real review there is no list, and a discrepancy the first pass missed and the member did not catch is not a cost in the table at all; it is a residual risk, whose size depends on what was missed and what follows from it, and the table cannot price it without assumptions the committee would have to state. What the trial does show is how much each setup left for the member to find, which is one piece of evidence about that risk.

Two published accounts give this structure at organizational scale, each within its own scope. Uber's engineering post from August 2026, summarized in the site's Intel collection, breaks the cost of agent-written code into factors multiplied together: how many people use the agents, how many sessions each runs, how many turns per session, how many model requests per turn, how many tokens per request, and the price per token. It then reports reductions from choosing models by measured performance on its own tasks, from running sequences of tool calls as scripts, and from loading tool descriptions on demand, and it says the savings are specific to Uber's environment. 9 10 The factors are the useful part: each is a place cost can be counted, and a comparison that counts only one of them is incomplete. A second account, from a Spotify engineer in September 2026, describes routing rote work such as file inventories and repetitive edits to cheaper models through an internal portal and keeping difficult reasoning with the main model. The original post was unavailable when this lesson was written; the site's collected summary reports a ninety percent reduction and states that the figure is the main model's token usage on a small codebase, not a total cost. 11 Read with that scope, the routed work still ran on some model and cost something, and the summary itself notes that summaries lose information and delegation adds latency.

The misconception both accounts correct, read with their scopes, is that fewer main-model tokens means a lower total bill. It may. It also may not, when the routed work costs more than expected, when the summaries it produces cause misses that a person then finds, or when the added latency has a cost of its own. The only way to know is to count the whole, which is what the table does for the committee's afternoon and what Uber's factors do for a company.

Carry this forward

  • Read a comparison table quality first, including unplanted findings checked by hand and whether the totals are right, then runs including the failed ones, then elapsed time, then review time, then the observed data path. Each is a separate column because each is a separate cost or constraint.
  • Total work cost is everything the task consumed to reach an accepted report: every run, the elapsed waiting, the review and correction time, the committee's own machine costs, and the residual risk of what the member did not catch. In this trial the completed call's price was a small part; that is an observation, not a rule.
  • Misses the member corrected are counted as review time, not again as later loss. Misses no one catches are residual risk that a table cannot price without stated assumptions; what a setup left for the member to find is the indicator.
  • Local inference kept the model's computation on the laptop and a web-search tool still sent one query out. What the trial observed leaving is not what the setup guarantees; logs, telemetry, backups and shell access are separate routes to check.
  • Fewer main-model tokens do not by themselves mean a lower total bill. Published reductions describe one component in one environment; the routed work, the failures, the review and the latency are all still in the total.

Lesson check

In the authored table, setup A's completed run cost 6.40 units and its failed run 2.10. A member argues that only the 6.40 should be compared with B's 1.10, because the failed run produced nothing. What does this lesson say?

Choose an answer for each question.

This is a practice check. Answers reset when you leave this page. Results are not saved and do not place you in another module or award a qualification.

The table is read. The committee still has to choose, and to write the choice down in a form that says why, what is still open, and when to look again.

Sources · Reviewed 2026-09-29
  1. 9. Uber Engineering: How Uber measures and cuts the cost of agent-written code (Uday Kiran Medisetty, 2026-08-27)
  2. 10. VibeLeaderboard Intel: How Uber measures and cuts the cost of agent-written code
  3. 11. VibeLeaderboard Intel: Spotify's Backstage Portal cut my Claude Code token usage by 90% (Dimitri Mazmanov, 2026-09-03)

Sources support nearby explanations. Product details can change. Scenarios and model results are authored illustrations unless a lesson explicitly identifies a recorded execution. Module 7 identifies its offline code checks.