Intel / article

Counting the Bandages: On measuring the time that AI saves

Source: arxiv.org
Author: Solomon Shalom Lijo
Date: Published Oct 28, 2026

Why it matters

Clock time alone overstates AI savings if defects escape. The paper reanalyzes METR's developer study to show the slowdown estimate shifts from 19% to 34% depending on the estimand, and offers a break-even defect cost for decisions.

Key takeaways · AI-distilled

In the paper's baseline checking model, a worker's escaped-defect rate does not depend on model quality until the model is good enough to skip checking, and checking time grows with the logarithm of the model's error rate.
The author's reanalysis of METR's public data finds the same 246 issues support a 19% or 24% slowdown depending on the estimand, 34% without adjusting for a chance imbalance, and that AI helped on somewhere between 3% and 82% of the issues.
Restating GDPval's results with stated bounds, the author finds that in its review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x.
In a small pre-specified test where AI reviewers checked AI-drafted briefs built from post-training-cutoff SEC filings, reviewers told the drafter's true error rate checked more as it rose, but all-or-nothing, switching near the prompt's cost break-even point.
Reviewers told the drafter erred ten times less often than it did checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none. The author reads this as faster and worse, entirely through reduced checking effort.

Terms in this piece · Glossary

calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

Read the source arxiv.org