11 LLM evaluation methods AI engineers must know: (bookmark this) Two eval…
- Source
- x.com
- Date
11 LLM evaluation methods AI engineers must know: (bookmark this) Two eval metrics can rank the same two models in opposite orders, and neither one is wrong. A model that paraphrases the reference can score near zero on BLEU and near the top on BERTScore for the exact same output. Neither metric is wrong because one is measuring wording and the other is capturing meaning. This is why LLM evaluation is fragmented into several methods, depicted in the visual below and grouped by what each one assumes: > Reference-based (ground truth exists): - BLEU - ROUGE - BERTScore > Judge-based (no ground truth): - G-Eval - LLM-as-Judge - LLM juries > Human and deterministic: - Human eval - DAG > Built for agents: - Trajectory accuracy - Multi-turn eval > Run as a gate: - Safety eval To use them in practice, most of these metrics are already implemented in Opik, which is open source (20k+ stars) and runs them over traced production data. You can start using them in a few lines of code. GitHub repo: https://t.co/vahjkkfJCt (don’t forget to star it ⭐️) That said, metrics only point at the failing case. The rest of the work is still done manually, like inspecting the trace to…
Context
Two scores can disagree because they measure different kinds of similarity. Avi Chawla's post illustrates this with a paraphrase: an answer may preserve meaning while changing the wording used in the reference answer.
BLEU compares matching words and word sequences against reference translations. BERTScore compares learned representations of words in , allowing similarity beyond exact wording. A score therefore needs to be interpreted through what its method rewards. Closeness to a reference, by either measure, is a narrower question than whether an completed the user's task.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Checking sign-in…
Loading comments…