Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

Hamel Husain and Shreya Shankar, who teach a course on AI evaluations, tell Lenny's Podcast that the first step is reading real conversation logs, not writing tests. Husain's walk-through uses logs from an apartment-leasing assistant. For each logged interaction a person writes a short note on the first thing that went wrong, for example that it should have handed off to a human or offered a virtual tour that does not exist, then moves on. They suggest about 100 examples, or until new notes stop appearing.

The notes are then grouped into failure categories, with an AI tool helping, and counted to see which problems are most common. Only problems that fixing the prompt does not resolve get an automated check, often a judge model giving pass or fail on one narrow failure. They say the judge must be compared with human labels, since 90 percent agreement can come from passing everything when errors are rare, and that the note-taking should stay human because an AI lacks the product context to spot such errors. This is their recommended practice, not a measured comparison.