AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
Source
arxiv.org
Author
Mehil B Shah, Mohammad Masudur Rahman, Foutse Khomh
Date
Why it matters
Gives a reproducible set of real faults from agentic and LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →-based systems to test debugging agents against, instead of relying on synthetic or traditional-software bugs.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleCompound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production IncidentsRudrendu Kumar Paul, Sourav Nandy
articleToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling AgentsYiShan Zheng, Yuan Wu, Yi Chang
articleSWEeper-Bench: Can Agents Discover Bugs in Interactive Software?Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu