Hamel Husain is a machine learning engineer and educator focused on evaluating AI products. He previously worked at Airbnb and GitHub, led the team behind CodeSearchNet, and teaches teams how to inspect failures and measure whether their systems improve.[1][2][3]
Start with Your AI Product Needs Evals →
GitHub 
X Career
Husain’s work spans industry machine learning, open-source tools and education. His personal site lists Airbnb and GitHub among earlier employers. In his evaluation essay he identifies CodeSearchNet as a project his team created. The research benchmark asks systems to retrieve relevant code from natural-language requests, a problem in which a developer’s description and the code’s actual vocabulary can differ substantially.[1][2][3]
Contributions
The 2019 CodeSearchNet paper, which Husain co-authored, released a corpus of functions across six programming languages and expert relevance judgments for search queries. Those judgments let researchers test whether retrieved code answers a request, rather than measuring only whether text strings match. The paper also supplies baseline methods and documents how its corpus was assembled, making competing search systems easier to compare.[3]
Ideas
Husain argues that AI-product quality needs task-specific evaluations in addition to general model benchmarks. His case study of the Lucy real-estate assistant connects tests, trace inspection and behavior changes in an iterative development process. His LLM-as-a-Judge guide then explains how to align automated judging with a domain expert: collect realistic examples, study critiques and validate the judge against human decisions. This gives teams a way to turn vague dissatisfaction into identifiable failure categories and measurable changes.[2][4]
Tools, writing & talks
Writing, research & talks
Start here
Your AI Product Needs Evalshamel.devA case study connecting product-specific tests, debugging and repeated improvements to an AI assistant.
Read the product-evaluation case study first if you have an AI feature but cannot tell whether changes help. Familiarity with software tests is useful; begin by collecting actual failures before choosing an automated judge.
Author[2]- How To Use Decision Models for Evalshamel.dev
Hamel Husain's team shows how to tune thresholds and calibrate scores for decision models (Jev, Decider, d1, Clef, OpenAI Decisions API), comparing cost and accuracy against LLM judges on 100 apartment-leasing agent traces.
Catalog workOriginal source - Claude’s new auto eval toolhamel.dev
Hamel Husain and Isaac Flath livestream Anthropic's new eval-building and hill-climbing commands on real leasing-assistant traces, and flag where the workflow skips error analysis and asks for judgments without enough context.
Catalog workOriginal source - Do Automated Evals Work?hamel.dev
We compared 100 human annotated traces against automated eval systems. Here's what we found.
Catalog workOriginal source - The Revenge of the Data Scientisthamel.dev
An argument that the data scientist's unusual mix of statistics and engineering, once called the sexiest job of the century and then written off, is exactly what the current moment rewards.
Catalog workOriginal source - Evals Skills for Coding Agentshamel.dev
A set of skills for AI product evals, built to guard against the mistakes seen across 50-plus companies and 4,000-plus students, now that coding agents instrument applications and run the experiments themselves.
Catalog workOriginal source - Why I Stopped Using nbdevhamel.dev
Hamel Husain on why he stopped using nbdev, and the more general point that a tool feels like a secret weapon only while it matches how you happen to think.
Catalog workOriginal source - [Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankaryoutube.com
Hamel Husain and Shreya Shankar discuss why AI products stall beyond demos and how error analysis and systematic evals provide a path for measuring and improving LLM applications.
Catalog workOriginal source - Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankaryoutube.com
Two eval experts lay out a lightweight process for evaluating AI products: open-code real traces with a single trusted domain expert, avoid committee overhead, and treat evals as an actionable improvement loop.
Catalog workOriginal source - Selecting The Right AI Evals Toolhamel.dev
Why the question of the best evals tool resists a direct answer: teams focus on tools instead of process, expecting an off-the-shelf solution, and the tools change faster than any comparison survives.
Catalog workOriginal source Curated answers to the questions that came up most while teaching AI evals to 700-plus engineers and product managers, offered as sharp working opinions rather than universal rules.
Catalog workOriginal sourceMost AI teams invest in architecture before measurement. A field guide to improving AI products that starts by asking how you would know whether any of it works.
Catalog workOriginal sourceA practical guide to using a model as a judge, drawn from setting up evaluation systems at more than 30 companies, and the mistakes teams repeat when they try it.
Catalog workOriginal sourceMastering LLMs, an open course of workshops and talks from 25-plus practitioners covering evals, retrieval-augmented generation and fine-tuning, with an emphasis on applying prior art from search and recommender systems.
Catalog workOriginal sourceAdversarial validation as a simple way to detect drift: train a classifier to tell two datasets apart, and if it succeeds, your evaluation data no longer resembles what the model sees in production.
Catalog workOriginal source- Your AI Product Needs Evalshamel.dev
Hamel Husain's argument that unsuccessful LLM products almost always share one root cause, the absence of a robust evaluation system, drawn from five years of building and consulting on them.
Catalog workOriginal source - Is Fine-Tuning Still Valuable?hamel.dev
Hamel Husain's answer to growing disillusionment with fine-tuning: it remains valuable in specific situations, and the case against it usually generalises from the wrong ones.
Catalog workOriginal source - Using LLM-as-a-Judge for Evaluationhamel.dev
A practical guide to validating automated judgments against domain-expert critiques.
Author[4] - CodeSearchNet Challengearxiv.org
Research dataset and evaluation task for retrieving code from natural-language requests.