SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?
Source
arxiv.org
Author
Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu
Date
Why it matters
Most benchmarks hand agents a known bug. This one requires discovering interaction-dependent bugs by testing in a browser, and the top frontier AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → solves only 59%.
Key takeaways · AI-distilled
Stage-by-stage trajectory analysis puts the main bottleneck at identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure, or trigger it and fail to recognize it.
Tasks come from real bug fixes in 200 widely used open-source web apps, and prompts are open-ended, for example 'Find and fix issues in Focalboard's board sharing and membership,' with no bug report supplied.
Repairs are judged by an agentic verifier on what a user would observe in the browser rather than by unit tests, since these bugs are hard to detect from code alone.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleCheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
articleFramework and Benchmark for Code-Driven Agentic Testing in Web DevelopmentBin Hong, Zhenchao Zhang, Jiyuan He, Kai Zhang, Zhenya Huang
articleWebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution HarnessYun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang