Rating the Raters: Rasch Measurement Theory for LLM Evaluation

An AI system's rating can reflect its own threshold for flagging content as well as the content itself. In Rating the Raters, researchers use Rasch measurement models to separate those contributions, placing comments, survey questions and raters on a shared scale so that a rater's habits can be estimated apart from what is being rated.

In their nine-model hate-speech study, every model sat below the median human rater on the paper's severity scale, meaning it flagged hatefulness more readily. Reversing the order of the survey questions also shifted thresholds significantly for four of the models. The authors do not advocate replacing human annotators; they argue that high agreement scores can hide differences in how a model uses the rating scale, which is why this kind of per-rater check is worth running.