In-House LLM Serving at Netflix

A rare account of running the whole LLM stack in-house rather than calling a hosted API — engine choice, packaging, API surface, and enforcing output constraints — including which trade-offs only became visible under production traffic.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

• blogMAPS: Netflix’s Multimodal Asset Personalization at Scale
• blogGenRec: Towards LLM-Native Recommendation at Netflix
• blogGenPage: Towards End-to-End Generative Homepage Construction at Netflix

• articleSlack AI: The Path to Multi-CloudShaurya Kethireddy
• articleNetflix PRS 2024 - Applying LLMs to Recommendation ExperiencesEugene Yan
• articleHow We Leveraged Vllm To Power Our Genai ApplicationsLinkedIn Engineering

Checking sign-in…

Loading comments…