Vibeleaderboard
← All Intel
Intel / article

JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

Source
arxiv.org
Author
Erlis Lushtaku, Bora Kargi, Ali Elganzory, Fabio Ferreira, Alejandro R. Salamanca, Julia Kreutzer, David Salinas
Date

Context

JudgeArena makes the model doing the grading an explicit part of an . It brings several benchmarks into one framework, lets researchers swap the judging model and records configuration details. That makes it easier to test whether a ranking changes because the candidate improved or because the judge, prompt or evaluation setup changed. Framework description.

The authors report that tuned open-model judges can match or exceed closed-model judges on human-preference datasets. They also combine existing human annotations with automated judgments to estimate arena-style rankings. These are estimates of preference-based evaluation, so their agreement with people should not be read as a separate demonstration of factual accuracy or safety. Reported evaluation.

Terms in this piece · Glossary
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
  • LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Key quotes

“LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol.”

“Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models.”

“Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.”

Recommended reads
Comments

Checking sign-in…

Loading comments…