Hamel Husain

Programmers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “ secret weapon .” DHH described Ruby as “ a magical glove that just fit my brain perfectly .” Pieter Levels ships million-dollar products with vanilla PHP and jQuery . These declarations aren’t about the languages themselves. They’re about developers finding tools that fit how they think. When the environment clicks, you move fast. I had that experience with nbdev , a development environment for literate progr

Most AI teams focus on the wrong things. Here’s a common scene from my consulting work: AI TEAM Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for… ME [Holding up my hand to pause the enthusiastic tech lead.] “Can you show me how you’re measuring if any of this actually works?” … Room goes quiet This scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can’t tell me if their chang

Today, I’m publishing evals-skills , a set of skills for AI product evals 1 . They guard against common mistakes I’ve seen helping 50+ companies and teaching 4,000+ students in our course . Why Skills for Evals Coding agents now instrument applications, run experiments, analyze data, and build interfaces. I’ve been pointing them at evals. OpenAI’s Harness Engineering article makes the case well. They built a product entirely with Codex agents — three engineers, five months, ~1 million lines of c

We compared 100 human annotated traces against automated eval systems. Here's what we found.

Today, we are releasing Mastering LLMs , a set of workshops and talks from practitioners on topics like evals, retrieval-augmented-generation (RAG), fine-tuning and more. This course is unique because it is: Taught by 25+ industry veterans who are experts in information retrieval, machine learning, recommendation systems, MLOps and data science. We discuss how this prior art can be applied to LLMs to give you a meaningful advantage. Focused on applied topics that are relevant to people building

Motivation I started working with language models five years ago when I led the team that created CodeSearchNet , a precursor to GitHub CoPilot. Since then, I’ve seen many successful and unsuccessful approaches to building LLM products. I’ve found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems. I’m currently an independent consultant who helps companies build domain-specific AI products. I hope companies can save thousands of dol

Earlier this year, I wrote Your AI product needs evals . Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in Data Ever spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs: Too Many Metrics : Creating numerou

img.img-fluid { border: 1px solid rgba(255, 255, 255, 0.25); border-radius: 4px; } Is the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.” 1 In tech, data scientist roles were often among the best paid. 2 The job also demanded an unusual mix of skills: Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. — JosH100 ( @josh_wills ) May 3, 2012

Over the past year, I’ve focused heavily on AI Evals , both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately. Having used many of the popular eval tools, I can genuinely say t

This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. For a guided path through the rest of our evals work, use the AI evals topic hub . 👉 Want to learn more about AI Evals? Check out our AI Evals course . It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈 Listen to

Here is my personal opinion about the questions I posed in this tweet : There are a growing number of voices expressing disillusionment with fine-tuning. I'm curious about the sentiment more generally. (I am withholding sharing my opinion rn). Tweets below are from @mlpowered @abacaj @emollick pic.twitter.com/cU0hCdubBU — Hamel Husain ( @HamelHusain ) March 26, 2024 I think that fine-tuning is still very valuable in many situations. I’ve done some more digging and I find that people who say that

For years, I’ve relied on a straightforward method to identify sudden changes in model inputs or training data, known as “drift.” This method, Adversarial Validation 1 , is both simple and effective. The best part? It requires no complex tools or infrastructure. Examples where drift can cause bugs in your AI: Your data for evaluations are materially different from the inputs your model receives in production, causing your evaluations to be misleading. Updates to prompts, functions, RAG, and simi