Rating the Raters: Rasch Measurement Theory for LLM Evaluation
Source
Pratik S. Sachdeva, Nathan Boudol
Author
Pratik S. Sachdeva, Nathan Boudol
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
Why it matters
If you use LLMs as judges or annotators, their scores carry measurable severity and calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → biases. This gives a psychometric way to diagnose those biases instead of trusting raw agreement with humans.
articleSelf- and Other-Labels Induce Bidirectional Bias in LLM JudgesSongeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
articleInter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextHaoyuan Li, Snigdha Chaturvedi
articleThe Limits of Automatic Evaluation of Creativity in Large Language ModelsAlessandro Tutone, Giorgio Franceschelli, Mirco Musolesi