[Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Source
youtube.com
Author
Latent Space
Date
Context
An AI model that grades another model's answers needs its own evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →. In this discussion of AI evaluations, Hamel Husain, who helps teams improve AI applications, recommends comparing an automated grader's decisions with a domain expert's judgments on examples from the actual application. A strong general benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → score does not establish that the grader recognizes the failures that matter in a particular product.
That comparison gives automated scoring a concrete purpose: extending a checked human standard across more outputs. Without it, a team can accumulate reassuring scores while measuring something different from the quality its users need.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
videoHow to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents
videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
