BenchMIRT: What are LLM benchmarks actually measuring?

A benchmark labeled as a safety test can also measure whether a model understands the question. Ai2's BenchMIRT analysis examines individual answers across models to estimate which underlying abilities predict success, rather than assuming every question measures the benchmark's advertised purpose.

Across 100 models and 16 benchmarks, the analysis recovered dimensions associated with reasoning and safety. Questions in BBQ, a social-bias benchmark, tracked reasoning much more strongly. A poor score can therefore reflect difficulty interpreting the scenario as well as biased behavior. This finding concerns the selected evaluations and models released by March 2025; it does not establish that every AI capability reduces to those two dimensions.