TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
Source
arxiv.org
Author
Kazuki Egashira, Niels M\"undler-Sasahara, Mark Niklas M\"uller, Martin Vechev
Date
Context
Tests that pass on working software can still miss broken behavior. TestSuiteBench measures that gap across 165 applications by asking coding agents to generate tests from detailed specifications. Researchers then run those tests against deliberately altered implementations, each designed to violate a requirement.
The strongest tested AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → detected 55.5% of those faulty variants, even though 96.1% of its generated tests passed on the reference implementations. Passing establishes that a test accepts expected behavior; detecting a faulty variant establishes that it rejects a particular defect. The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → exposes a substantial gap between those two abilities.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleGrounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test GenerationMichele Tufano, James McClure, Jos\'e Cambronero, Runxiang Cheng, Sherry Y. Shi, Renyao Wei, Dorothy Chen, Franjo Ivan\v{c}i\'c, Livio Dalloro, Pat Rondon
articleWebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution HarnessYun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang
articleCheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan