Using LLM-as-a-Judge For Evaluation: A Complete Guide
Source
Hamel Husain
Author
Hamel Husain
Date
Key takeaways · AI-distilled
Pick exactly one principal domain expert and optimize for their judgment alone. Several reviewers will disagree about what 'good' means, and averaging them yields an evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → nobody trusts; one expert gives a consistent bar and a stake in the outcome.
The common failure is the developer appointing themselves domain expert, or promoting their boss into the role. You end up optimizing against an opinion no user ever validated.
Build the eval set along dimensions you choose in advance, such as features, scenarios and personas, then generate inputs for combinations of them. Coverage becomes deliberate instead of whatever production happened to log.
Define scenarios by the situation the system faces, not by how it answered: 'user has three open orders', 'order number doesn't exist', 'user omitted the return reason'. That keeps the test set independent of any one model's behavior.
Use an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → to synthesize the user inputs only, never the responses or internal system behavior. Seed those prompts with real API and database state so each generated case actually triggers the scenario you meant to test.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
A step-by-step methodology for building trustworthy LLM-as-a-judge systems, replacing arbitrary 1-5 scoring with 'Critique Shadowing' that anchors evals to a single domain expert's judgment. Useful if you're drowning in unvalidated eval metrics and can't tell whether your AI product actually works.
Key quotes
“If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong.”
“I can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing.”
“I would go as far as saying that creating a LLM judge is a nice “hack” I use to trick people into carefully looking at their data!”
“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”
