11 LLM evaluation methods AI engineers must know:
(bookmark this)
Two eval metrics can rank the same two models in opposite orders, and neither one is wrong.
A model that paraphrases the reference can score near zero on BLEU and near the top on BERTScore for the exact same output.
Neither metric is wrong because one is measuring wording and the other is capturing meaning.
This is why LLM evaluation is fragmented into several methods, depicted in the visual below and grouped by what each one assumes:
> Reference-based (ground truth exists):
- BLEU
- ROUGE
- BERTScore
> Judge-based (no ground truth):
- G-Eval
- LLM-as-Judge
- LLM juries
> Human and deterministic:
- Human eval
- DAG
> Built for agents:
- Trajectory accuracy
- Multi-turn eval
> Run as a gate:
- Safety eval
To use them in practice, most of these metrics are already implemented in Opik, which is open source (20k+ stars) and runs them over traced production data. You can start using them in a few lines of code.
GitHub repo: https://t.co/vahjkkfJCt
(don’t forget to star it ⭐️)
That said, metrics only point at the failing case.
The rest of the work is still done manually, like inspecting the trace to…
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Choosing the wrong evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → metric silently inverts model rankings, which is a common and expensive mistake.
Read the full post on X ↗
