XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?
Source
Yanfei Hu Fleischhauer, Alona Zharova, Nadja Klein, Stefan Feuerriegel
Author
Yanfei Hu Fleischhauer, Alona Zharova, Nadja Klein, Stefan Feuerriegel
Date
Terms in this piece · Glossary
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Shows LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judges can replace costly human panels for scoring explainability output quality with strong correlation to human ratings, letting teams scale XAI evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →.
Key quotes
“Human validation shows a strong positive association between LLM-generated and human ratings (Spearman's rho=.693, p<.001).”
“Together, LLM-based evaluations can capture systematic differences in XAI explanation quality and provide a scalable and reproducible framework for comparative assessment of XAI explanations.”
articleRating the Raters: Rasch Measurement Theory for LLM EvaluationPratik S. Sachdeva, Nathan Boudol
articleLitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review PlatformRuotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
articleThe Limits of Automatic Evaluation of Creativity in Large Language ModelsAlessandro Tutone, Giorgio Franceschelli, Mirco Musolesi