Vibeleaderboard
← All Intel
Intel / article

Making OpenClaw CI faster and 95% cheaper · Vincent Koc

Source
vincent.build
Date

Context

The 95% in Vincent Koc's account of OpenClaw's continuous integration (CI), the automated checks that run on every proposed code change, is a drop in the estimated value of compute between two different weeks, not a saving on a bill or on identical work (Vincent Koc). The machines that run the checks logged 158,546 hours in the week of 17 to 23 September and 21,395 in the week ending 13 November, 86.5% less. Priced at the published rates of Blacksmith, the company that supplies those machines, the first week's usage is worth about $1.27 million over 30 days and the second about $57,324, which is the 95.5% in the headline. Blacksmith sponsored the capacity, so OpenClaw did not pay those amounts. Activity also fell: jobs were down 79.7%, and commits to the main repository about 44%. Koc says the workload changed, so no single fix earns either reduction.

What the article does show is where the work went. Agents had been writing code and tests faster than the checks could keep up, and Koc's first step was deleting tests that only restate the code they test or duplicate a stronger check; in OpenClaw's core source, test files shrank by about 418,000 lines net between 28 September and 13 November, while a guard now makes large additions of test code carry a recorded justification. The full suite stopped running after every merge to the main branch; broader checks of main now run hourly on a schedule, and full release validation on a schedule of its own. Then, around 6 and 7 November, OpenClaw switched on test selection: a deterministic planner, ordinary code rather than a model, picks the tests that own or import the files a change touches and runs more when it cannot tell. In three example pull request runs it executed 15 of 4,018 test entries, 16 of 4,480 and 97 of 6,895. Estimated daily cost fell from about $7,738 on 5 November to $1,507 on 7 November, but Koc notes that 5 November was already about 82% below the September peak's daily average and calls the step chronology, not proof of what selection alone saved. Learned models that predict which tests will fail only record their guesses; they cannot skip anything.

Cheaper did not clearly mean faster. In Koc's samples of successful pull request runs, the median wait for CI was 6 minutes 41 seconds in June, 12 minutes 10 seconds in September and 9 minutes 18 seconds in a partial sample on 4 November, all before selection rolled out, and he says any improvement in the time from opening a pull request to merging it is not yet established. One matched job makes the tension concrete: moving the same 240 test files to a smaller machine cut the estimated cost by 41.7% and made the job 16.5% slower. The supported lesson is narrower than the headline: in a codebase where agents generate much of the code and tests, most of the saving came from running less work, and wait time has to be measured separately because a lower bill does not show that changes land sooner.

Key takeaways · AI-distilled
  • Vincent Koc reports OpenClaw's estimated CI compute value fell 95.5%, from about $1.27M to $57K per 30 days at Blacksmith's published prices, with runner-hours down 86.5%. Blacksmith sponsored the compute, and he notes the workload also changed, so no single fix owns the drop.
  • The test cleanup asks each test to name the behavior it protects and a believable bug that would break it, keeping one strong owner per contract. A test-line budget then makes suite growth explicit: larger additions need a recorded, justified allowance.
  • Adaptive selection maps a diff to owning tests and to tests of importing callers with deterministic code, not an AI scheduler. Unknown or missing dependencies widen the run. Three sample PR runs executed 15, 16 and 97 entries out of inventories of 4,018 to 6,895.
  • Learned test selection runs only in shadow mode: the model records its proposed subset, then every test still runs so misses can be counted. Koc notes zero misses across N faulty changes only bounds the miss rate near 3/N, so skip authority needs far more evidence.
  • Cheaper is not always faster: the same 240 test files cost an estimated 41.7% less on a 16-class runner but took 16.5% longer. Koc also found over-sharding repeated setup; rebalancing 18 planned jobs into 20 cut the slowest job from 403s to 300s.
Read the source vincent.build
Recommended reads
Comments

Checking sign-in…

Loading comments…