Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)
- Source
- magazine.sebastianraschka.com
- Author
- Sebastian Raschka, PhD
- Date
Context
A model's accuracy and its popularity with users answer different questions. Sebastian Raschka's evaluation tutorial separates checking answers against known solutions from judging the quality of a response.
Multiple-choice tests score selected answers. Verifiers can check free-form outputs, such as a mathematical result, when correctness is reliably testable. Preference rankings instead aggregate choices between responses, which can reflect writing style as well as substance. An applies a supplied grading rubric. Choosing an therefore starts with deciding what success means: a preferred explanation and a verified answer provide different evidence about a model.
- The four eval methods fall into two families. Benchmark-based ones (multiple choice, verifiers) grade against a fixed answer key; judgment-based ones (leaderboards, judges) grade on someone's preference, whether human voters or another model.
- MMLU spans 57 subjects and about 16,000 multiple-choice questions, scored as plain accuracy: 14,000 right out of 16,000 is 87.5%. It measures knowledge recall the way a standardized test does.
- One MMLU number can be computed several ways: compare the generated answer letter against the key, or score the log-probability the model assigns to each choice. Two reported MMLU accuracies aren't automatically comparable.
- Prompt shape is part of the measurement. The prompt ends with 'Answer: ', trailing space included, to push the model toward emitting a single letter next. Few-shot examples (5-shot MMLU) were once needed; today's base models manage without.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
“One of the reasons why judges work so well is also that evaluating an answer is often easier than generating one.”
“Unfortunately, there is no single best method since, as we have seen, each comes with different trade-offs.”
“the best evaluation combines multiple areas. But ideally it also uses data that directly aligns with your goals or business problems.”
Checking sign-in…
Loading comments…