Vibeleaderboard
← All Intel
Intel / article

Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation

Source
arxiv.org
Author
Michele Tufano, James McClure, Jos\'e Cambronero, Runxiang Cheng, Sherry Y. Shi, Renyao Wei, Dorothy Chen, Franjo Ivan\v{c}i\'c, Livio Dalloro, Pat Rondon
Date

Context

An generated more effective tests when it first wrote down the code's behavioral contract: required inputs, promised outputs and undefined behavior. That intermediate description directed subsequent test generation toward boundaries the agent might otherwise overlook. The study evaluated this approach on 90 historical production bugs from Google.

Across five test-generation attempts per bug, estimated detection rose from 53.4% for the baseline to 63.2% with specification guidance. Detection required a test to fail on buggy code and pass on its repaired version. The result supports documenting behavior before writing tests, while measuring bug discovery rather than proving the code correct.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Recommended reads
Comments

Checking sign-in…

Loading comments…