When Can LLMs Replace Humans in A/B Tests?

Replacing people with model-generated responses changes what an A/B test measures. In Spotify's analysis of headline experiments, raw language-model predictions recovered only 39% of the observed human treatment effect. That percentage describes the estimated effect of changing headlines, rather than prediction accuracy.

Flexible calibration against historical human experiments brought estimates within the human benchmark's sampling uncertainty. But transfer requires the model to capture what matters about the change and the relationship between predictions and human behavior to remain stable. Historical agreement therefore supports a narrower use: estimating outcomes for sufficiently similar experiments, with human data still needed to establish and check that relationship.