When Can You Ship on Offline Evals Alone?
- Source
- engineering.atspotify.com
- Author
- Spotify Engineering
- Date

Context
A Spotify engineering post concludes that a team can ship a change to an AI feature on offline alone only under strong assumptions that are hard to verify and can change over time, and Spotify itself still pairs its evals with online experiments (Spotify Engineering). An offline eval runs a new prompt or model against a fixed set of sample prompts before any user sees it, often with another grading the answers, and takes hours. An A/B test gives the new version to a random group of real users and compares their behavior with users on the old one, and often takes days or weeks. The question is when the fast test can stand in for the slow one without much risk of shipping something that hurts the metric users actually experience.
The answer depends on a measured relationship, not a hunch. A team needs a history of past changes that each have both an eval result and an A/B result, all of the same kind, so it can learn how well the eval predicts the online outcome. Both numbers are noisy, and the post shows that ignoring that noise makes a strong link look weak: in the paper's working example, a naive estimate says the eval explains 24% of the variation in online results when the true figure is 60%. With the relationship measured, the team can set a gate in which a clearly positive eval result ships, a clearly negative one is killed, and anything in between still goes to an A/B test. The weaker the relationship, the stronger the eval evidence has to be before it decides anything alone.
Three things can quietly break that gate. If the history mixes different kinds of changes, pooling them can show a strong link even when the eval predicts nothing within any single kind. An LLM judge that is biased the same way for old and new versions does no harm, because the bias cancels out, but a judge whose bias depends on which version it grades can make a worse version look better, a risk the post expects when teams compare underlying models, since judges can favor their own model's output. Once the gate is running, changes it ships or kills no longer get A/B tests, so the team loses the evidence that would show the relationship drifting. Spotify's fix is an audit pool, a small random share of those changes that still get A/B tested, at the cost of fewer eval-only decisions. The post makes the same argument Spotify made in August about replacing human outcomes with LLM predictions: A/B tests bound risk by design, while the shortcut holds only by assumption.
- Spotify frames shipping on offline evals alone as bounding the risk of harming the online metric. That requires paired eval and A/B results from changes sharing one underlying relationship; pooling unlike changes can show a strong link that exists within none of them.
- Noise in both eval scores and A/B estimates weakens the observed offline-online correlation. In Spotify's worked example, a naive estimate says the eval explains 24% of online variation when the true figure is 60%, so noise can hide a strong proxy.
- Spotify's proposed gate ships on sufficiently positive eval effects, kills on sufficiently negative ones, and sends the middle to an A/B test. The weaker the eval-to-online relationship, the stronger the eval evidence needed before skipping a test.
- An LLM judge's bias cancels in the treatment-control difference unless it depends on the arm, such as a judge favoring its own model's outputs when teams compare underlying LLMs. Arm-dependent bias can even flip the sign, making a worse version look better.
- Once an eval gate is live, ship and kill decisions stop producing A/B ground truth, so drift goes unseen exactly where the gate decides. Spotify proposes an audit pool that still A/B tests a small random share of gated changes, at the cost of fewer test-free decisions.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleAI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity- articleSpotify’s Backstage Portal cut my Claude Code token usage by 90%
articleCoding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify
articleIndexing the Data Lake for Online Point Queries
Checking sign-in…
Loading comments…


