A cheap judge can match a frontier judge on most multilingual evaluation tasks, but it wrongly accepts more incorrect answers in low-resource languages. Use its probabilities for confidence-based routing.
Key takeaways · AI-distilled
Jev is a decision model that outputs judgment probabilities directly instead of generating text; the authors test it against generative LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judges on four human-annotated benchmarks covering up to 47 language varieties.
The comparison spans binary decisions, pairwise comparisons and graded ratings, and the authors put Jev's evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → cost at roughly 1/31 to 1/61 of GPT-6.1's.
The authors find Jev's probability outputs help identify judging errors across tasks, and base their recommendations on evaluation instructions, confidence-based selection and model routingSending each request to a model chosen by the difficulty of the task, rather than using one model for everything.Full definition → on these findings.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
model routing — Sending each request to a model chosen by the difficulty of the task, rather than using one model for everything.
