Hamel Husain
15 Intel
Hamel Husain is a machine learning engineer and educator with more than two decades of experience in applied AI, machine learning, and data science, including prior roles at Airbnb and GitHub, where he contributed to early LLM-based code-understanding research (the CodeSearchNet benchmark) later used by OpenAI. He now works as an independent consultant helping companies evaluate and operationalize large language models, and co-teaches a widely taken course on AI evaluations for engineers and product managers.
Is this you? Sign in with X to claim this profile.
Intel
Hamel Husain on why he stopped using nbdev, and the more general point that a tool feels like a secret weapon only while it matches how you happen to think.
Most AI teams invest in architecture before measurement. A field guide to improving AI products that starts by asking how you would know whether any of it works.
A set of skills for AI product evals, built to guard against the mistakes seen across 50-plus companies and 4,000-plus students, now that coding agents instrument applications and run the experiments themselves.
Hamel Husain and Isaac Flath livestream Anthropic's new eval-building and hill-climbing commands on real leasing-assistant traces, and flag where the workflow skips error analysis and asks for judgments without enough context.
We compared 100 human annotated traces against automated eval systems. Here's what we found.
Mastering LLMs, an open course of workshops and talks from 25-plus practitioners covering evals, retrieval-augmented generation and fine-tuning, with an emphasis on applying prior art from search and recommender systems.
Hamel Husain's argument that unsuccessful LLM products almost always share one root cause, the absence of a robust evaluation system, drawn from five years of building and consulting on them.
Hamel Husain and Shreya Shankar discuss why AI products stall beyond demos and how error analysis and systematic evals provide a path for measuring and improving LLM applications.
A practical guide to using a model as a judge, drawn from setting up evaluation systems at more than 30 companies, and the mistakes teams repeat when they try it.
An argument that the data scientist's unusual mix of statistics and engineering, once called the sexiest job of the century and then written off, is exactly what the current moment rewards.
Why the question of the best evals tool resists a direct answer: teams focus on tools instead of process, expecting an off-the-shelf solution, and the tools change faster than any comparison survives.
Curated answers to the questions that came up most while teaching AI evals to 700-plus engineers and product managers, offered as sharp working opinions rather than universal rules.
Two eval experts lay out a lightweight process for evaluating AI products: open-code real traces with a single trusted domain expert, avoid committee overhead, and treat evals as an actionable improvement loop.
Hamel Husain's answer to growing disillusionment with fine-tuning: it remains valuable in specific situations, and the case against it usually generalises from the wrong ones.
Adversarial validation as a simple way to detect drift: train a classifier to tell two datasets apart, and if it succeeds, your evaluation data no longer resembles what the model sees in production.