Cutting our inference bill by 68% by being obsessive about optimization

Cloudflare cut the cost of its internal AI code reviewer from an average of $2.43 to $0.77 per review, and most of the money had been going to context the models had already seen or were about to see again, not to the reviews they wrote (blog.cloudflare.com). Before the work began, output tokens made up about 23% of spend, cache writes 43% and cache reads 34%. The reviewer runs on Anthropic models, whose prompt cache stores the start of a conversation, called a prefix, so a later request that begins with the same text can reuse it instead of paying to process it again. Writing to that cache costs 1.25 times the normal input price, reading costs 0.1 times, and an entry expires after five minutes unless something reads it. Every review session starts with 11k to 21k tokens of tool definitions and agent prompt that are identical across repositories, yet the first request of a session found a cached prefix only 14% to 26% of the time. The cause was that the reviewer's harness, OpenCode v1, put the shared prompt in the same block as the working directory and the date, with the only cache marker at the end of the block, so no repository could reuse another's entry. Splitting the block so the shared part ends on its own marker, and moving one reviewer's reference files into its prompt, took the cost per review from $2.47 to $2.02. Cloudflare later had three reviewers send identical first requests so that one writes the cache and the other two read it, with the second and third waiting up to 20 seconds for the first to start responding.

Cloudflare attributes about 40% of the savings to changes in the reviewer's code and the rest to the models it uses and how much effort they spend. Moving its Sonnet reviewers from Sonnet 5 to Sonnet 5.5, which cost the same per token, made runs 62% cheaper for code quality, 58% for performance and 47% for security, because the newer model used fewer tokens. The median review now takes 1m 32s instead of 4m 09s. Cloudflare notes that its before-and-after comparisons are not a controlled experiment.

The harder question is whether cheaper reviews miss problems. The code quality reviewer now posts 40% fewer critical and warning findings, but authors changed the flagged lines 59% of the time, up from 34%, so findings that led to a change stayed roughly level at 132 per 1,000 reviews against 145 before, within the margin of error. Model judges rating a blinded sample found important findings per review unchanged within overlapping intervals, while findings rated more severe than they deserved fell from 0.46 to 0.14 per review. That result holds for code quality at high effort, which the $0.77 figure assumes. The performance reviewer, still at medium effort, lost real findings, and the security reviewer's fall of about a quarter is not statistically significant. Cloudflare cautions that the judges are models, each group holds only 35 to 66 findings, and the latest window covers a single day.