How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Gives a concrete way to validate LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judges against human golden data and handle noise with power analysis, then confirm offline evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → in production A/B tests.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
videoStop Giving Your AI Agents API Keys - Kim Maida, Keycard & Mason Egger, Temporal
videoYour AI Agent Keeps Making the Same Mistakes - Sparsh Jain, CentralAgent AI
video7 Things Your AI Coworker Needs - Philipp Schmid, Google DeepMind
videoAI Agents Will Upend Investing Even More Than Coding - Frank Long, Coatue
