Intel / article

Using LLM-as-a-Judge For Evaluation: A Complete Guide

Source: hamel.dev
Author: Hamel Husain
Date: Published Oct 29, 2024

Context

An AI evaluator needs examples of what acceptable work means in the actual product. Hamel Husain's judge-building guide starts with a domain expert reviewing interactions, marking pass or fail and explaining the judgment. Those explanations expose requirements that a generic quality score leaves ambiguous.

The examples then inform the evaluator's instructions, and disagreements with the expert guide revisions. Husain also warns that overall agreement can hide poor detection when failures are rare. Measuring how many real failures the judge catches, and how often its failure flags are correct, makes that weakness visible.

Key takeaways · AI-distilled

Pick exactly one principal domain expert and optimize for their judgment alone. Several reviewers will disagree about what 'good' means, and averaging them yields an nobody trusts; one expert gives a consistent bar and a stake in the outcome.
The common failure is the developer appointing themselves domain expert, or promoting their boss into the role. You end up optimizing against an opinion no user ever validated.
Build the eval set along dimensions you choose in advance, such as features, scenarios and personas, then generate inputs for combinations of them. Coverage becomes deliberate instead of whatever production happened to log.
Define scenarios by the situation the system faces, not by how it answered: 'user has three open orders', 'order number doesn't exist', 'user omitted the return reason'. That keeps the test set independent of any one model's behavior.
Use an to synthesize the user inputs only, never the responses or internal system behavior. Seed those prompts with real API and database state so each generated case actually triggers the scenario you meant to test.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong.”

“I can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing.”

“I would go as far as saying that creating a LLM judge is a nice “hack” I use to trick people into carefully looking at their data!”

“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”
Phillip Carter

Read the source hamel.dev