Finding Flawed Questions in AI Benchmarks with Psychometrics

A benchmark score is only as trustworthy as its questions, and checking thousands of questions by hand takes expert time that is rarely available. Researchers at Stanford SAIL borrow a tool from test design to narrow the search. They treat each AI model as a test-taker and record whether it got each question right or wrong. If one average score is a fair summary of a model's ability, as most leaderboards assume, two patterns should hold. A model that answers one valid question correctly should be at least as likely to answer the others correctly, and models that score well overall should do better on each question. A question that breaks these patterns, for example one the strongest models keep missing, is flagged as a candidate for expert review. The flag is a lead for a human to check, not proof that the question is wrong.

The SAIL post names three kinds of flaw: ambiguous questions with more than one defensible answer, wrong answer keys, and graders that reject correct responses. One example of the last kind is a grader that accepts 4 but rejects 4.00. The team took right/wrong results from the HELM leaderboard for nine benchmarks, including math, medicine, safety, general knowledge and a Thai-language exam benchmark. Domain experts then reviewed the 50 questions ranked most suspicious in each benchmark. According to the paper, up to 84% of those 50 were confirmed flawed. That is the best result among the nine benchmarks, not an average, and it says how productive the shortlist was, not what share of all questions are broken. On GSM8K, a grade-school math benchmark that had already been cleaned up, the expert found 16 more invalid questions that the earlier revision missed. That earlier fix alone had moved DeepSeek-R1 from third lowest to second place on GSM8K.

Grading problems are the hardest to catch by reading questions, because a reviewer has to look at models' actual answers. For this, the researchers add a first pass by an LLM judge. It sees the question, the official key and several sample model answers, then labels the question valid or names the flaw and explains why, and a human checks its reasoning. The judge's test is a small one. ChatGPT o1 reviewed the first 100 GSM8K questions as graded in HELM. Human inspection found roughly 30% of them invalid, almost all because of grading, and the judge's flags were correct 98% of the time.

The method needs many varied models to work. Detection improved as more models, from more organizations and sizes, were added, and the authors recommend models from at least ten organizations and 60 to 80 models in total. The authors also say the statistics may not match human judgments of what counts as invalid; cultural ambiguity, for instance, can slip past purely numerical signals. No single signal caught every invalid question, and the current version handles only right/wrong scoring, with free-response formats left for future work.