Bootstrapping Conversational Recommendation Agents At Spotify: Synthetic Data Generation and Self-Improvement Loops
Source
Enrico Palumbo, Alexandre Tamborrino, Victor Ode, Ben Lacker, Adri\`a Casas Escoda, Jeremy Hopple, Marcus Better, James Leoni, Hugo Galv\~ao, Hugues Bouchard, Mounia Lalmas, Jos\'e Luis Redondo Garc\'ia, Abenezer Abebe, Ann Clifton, Anton Blomberg, Henrik Lindstr\"om, Dani Doro, Christine Doig Cardet
Author
Enrico Palumbo, Alexandre Tamborrino, Victor Ode, Ben Lacker, Adri\`a Casas Escoda, Jeremy Hopple, Marcus Better, James Leoni, Hugo Galv\~ao, Hugues Bouchard, Mounia Lalmas, Jos\'e Luis Redondo Garc\'ia, Abenezer Abebe, Ann Clifton, Anton Blomberg, Henrik Lindstr\"om, Dani Doro, Christine Doig Cardet
Date
Key takeaways · AI-distilled
To tune a conversational recommender before launch, Spotify turned single-turn prompts into realistic multi-turn synthetic conversations, giving the team an evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → set without real user traffic.
Its self-improvement loop pairs variance-based contrastive optimization with a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → that iteratively finds and fixes planning and tool-use errors, adding 8% quality on top of an already heavily tuned manual prompt.
In online A/B tests against a prior experience that only supported session refinement, the shipped agent lifted listening by 14% and weekly active users by 5% while cutting skip rate by 5%.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Spotify's approach to bootstrapping conversational recommendation agents, synthetic multi-turn dialogue generation plus a self-improvement loop for tool planning, is a concrete, transferable cold-start methodology for teams building agents without real usage data yet.
videoTeaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, SpotifyAI Engineer
articleCharacterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and ResponsesMahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander, Muhammad Bilal Zafar, Soumi Das
articleBeyond Surface Style: Aligning Multi-Turn User Simulators with Behavioral ConsistencyGeng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, Zheng Yang, Win-Bin Huang, Xiangyu Wu, Wenwu Ou