Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses
Source
Jannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu
Author
Jannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu
Date
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
Quantifies how often off-the-shelf LLMs falsely flag or miss requirements defects versus expert ground truth, showing teams should validate rather than trust default LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → requirements review.
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li
articleBeyond the Traceback: Using LLMs for Adaptive Explanations of Programming ErrorsAlexandru-Radu Moraru, Shreyan Biswas, Ujwal Gadiraju