Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses
Source
arxiv.org
Author
Jannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu
Date
Context
AI review can miss requirements problems that depend on engineering judgment. Researchers compared ten OpenAI and Anthropic models against expert assessments of two requirement sets. The best-performing Anthropic model detected a median of only 47% of expert-identified issues. Problems involving whether a requirement was necessary or correct were especially likely to escape detection, and newer model generations were not consistently better. These results support using the tested systems to assist expert review while retaining human judgment over substantive requirements decisions. A fluent assessment is insufficient evidence that the underlying specification has been checked adequately. Study abstract.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li