BenchMIRT: What are LLM benchmarks actually measuring?
Source
allenai.org
Date
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → averages mix capabilities: WildJailbreak's benign prompts track reasoning more than safety. Scoring prompts individually shows what an evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → actually measures and cuts how many questions you need.
videoBenchmaxxing: The Gap Between Benchmark Scores and RealityAI Engineer
articleBacktrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQsRuoxi Zhao, Maziar Raissi
articleThere Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile ItemsV. S. Raghu Parupudi