JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
- Source
- arxiv.org
- Author
- Erlis Lushtaku, Bora Kargi, Ali Elganzory, Fabio Ferreira, Alejandro R. Salamanca, Julia Kreutzer, David Salinas
- Date
Context
JudgeArena makes the model doing the grading an explicit part of an . It brings several benchmarks into one framework, lets researchers swap the judging model and records configuration details. That makes it easier to test whether a ranking changes because the candidate improved or because the judge, prompt or evaluation setup changed. Framework description.
The authors report that tuned open-model judges can match or exceed closed-model judges on human-preference datasets. They also combine existing human annotations with automated judgments to estimate arena-style rankings. These are estimates of preference-based evaluation, so their agreement with people should not be read as a separate demonstration of factual accuracy or safety. Reported evaluation.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
“LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol.”
“Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models.”
“Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.”
articleEvaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)Eugene Yan
articleChain-of-Models: Cross-Model Auditing for Bias-Robust LLM JudgesQian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Bingsheng He- blogThe Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scaleNetflix Technology Blog
Checking sign-in…
Loading comments…