Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody
Source
youtube.com
Author
Lenny's Podcast
Date
Why it matters
Explains that labs are bottlenecked on measuring success, so expert-written evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → drive model improvement. This clarifies where training signal for frontier models comes from.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoHow Block is becoming the most AI-native enterprise in the world | Dhanji R. Prasanna
videoWhy OpenAI is merging Codex and ChatGPT and the future of knowledge work | Andrew Ambrosino
videoAl Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)
videoOpenAI’s head of platform engineering on the next 12-24 months of AI | Sherwin Wu
videoRL Environments Explained: How AI Agents Learn Real-World Work | Brendan Foody, MercorSequoia Capital
podcastThe 100-person AI lab that became Anthropic and Google's secret weapon | Edwin Chen (Surge AI)Lenny Rachitsky