Netflix PRS 2024 - Applying LLMs to Recommendation Experiences
- Source
- eugeneyan.com
- Author
- Eugene Yan
- Date
Context
A generated summary can be checked without a human-written ideal answer. In Eugene Yan's 2024 Netflix workshop talk, the slides contrast two kinds of evaluation. Reference-based checks compare output with a gold reference, using n-gram metrics or similarity. Reference-free checks compare the output with its source document, for example through a natural language task, which tests whether the source supports a statement. The slides say this removes the bottleneck of needing golden references.
The published slides add that once such evals are built, they can be used as on new output. They also advise splitting an evaluation into binary dimensions, with factual consistency, information density and relevance as summarization examples, because binary labels and pairwise comparisons are easier to annotate than Likert (numeric rating) scales. The slides do not report how well any particular checker performs, and they warn that judges can be biased and inconsistent, so a checker would need validation before its decisions are trusted.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Checking sign-in…
Loading comments…