eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
Why it matters
If you're relying on an LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition → to guarantee product quality, this makes the case that the judge is only as good as your process — and lays out an evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →-driven development loop with hypothesis testing and output monitoring to actually catch regressions.
Key quotes
“Evals aren’t static artifacts or quick fixes; they’re practices that apply the scientific method, eval-driven development, and AI output monitoring.”
Eugene Yan
“Building product evals is simply the scientific method in disguise. That’s the secret sauce.”
Eugene Yan
“Ideally, we should have a 50:50 split of passes and fails that spans the distribution of inputs.”
Eugene Yan
“While automated evals help scale monitoring, they can’t compensate for neglect. If we’re not actively reviewing AI outputs and customer feedback, automated evaluators won’t save our product.”
Eugene Yan
“First, write some evals; then, build systems that pass those evals.”
