Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation
Source
arxiv.org
Author
Michele Tufano, James McClure, Jos\'e Cambronero, Runxiang Cheng, Sherry Y. Shi, Renyao Wei, Dorothy Chen, Franjo Ivan\v{c}i\'c, Livio Dalloro, Pat Rondon
Date
Context
An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → generated more effective tests when it first wrote down the code's behavioral contract: required inputs, promised outputs and undefined behavior. That intermediate description directed subsequent test generation toward boundaries the agent might otherwise overlook. The study evaluated this approach on 90 historical production bugs from Google.
Across five test-generation attempts per bug, estimated detection rose from 53.4% for the baseline to 63.2% with specification guidance. Detection required a test to fail on buggy code and pass on its repaired version. The result supports documenting behavior before writing tests, while measuring bug discovery rather than proving the code correct.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.