Intel / article

AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity

Source: engineering.atspotify.com
Author: Spotify Engineering
Date: Published Sep 16, 2026

Context

Music- company Spotify reports that faster software development exposed a verification bottleneck. Its merged changes rose from roughly 8,100 to 17,000 between August of last year and this August. In the incident reviews done so far, Spotify did not identify AI-written code as a material direct contributor, but it says the volume of change grew faster than some of its verification controls could adapt. One automated dependency upgrade passed its checks and still failed in production. For the automated fleet updates described in its earlier account, Spotify says it is strengthening safeguards, expanding rollback capacity and scheduling automated changes during the owning teams' working hours. In Spotify's account, faster implementation shifted the constraint to verifying releases and recovering from failures.

Key takeaways · AI-distilled

On June 24 a batch job competing with new episodes, a recent quality fix that raised per-episode compute, and a scheduling bug cutting throughput about 10% combined to delay episodes normally published within minutes by several hours.
Fix for the June 24 incident: end-to-end monitoring so failures surface before creators notice, a fixed scheduler, batch jobs moved to lower priority, added capacity, and reworked tiering so new uploads take precedence when capacity is constrained.
An automated dependency upgrade passed Fleet Management's safety checks but still broke production. Spotify responded by strengthening rollback capacity and scheduling automated changes during the owning team's working hours, not slowing automation.
Across reviewed production incidents, Spotify did not find AI-authored code to be a material direct contributor, but the increased volume of change outpaced verification controls (review, testing, rollout, observability), which the team is now strengthening.
Merged PRs more than doubled year over year (about 8,100 to 17,000), and quality/optimization work rose from 27% to 31% of that mix, so engineers put over twice the absolute effort into quality; rework rate stayed flat despite a wider industry churn rise.

Terms in this piece · Glossary

streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.

Read the source engineering.atspotify.com