Witness replay lets a coding agent extend a test generator while checking each addition against a recorded generation trace. The generators then run without further LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → calls and found real bugs.
Key takeaways · AI-distilled
Gungnir's coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → must submit a generation witness with each change: the database setup and semantic choices needed to produce a target SQL statement. A validator replays those choices through the modified generator and checks it reproduces that statement.
Every past witness is kept as a regression test, so later agent edits to the generator are checked against all previously validated features, so a new feature cannot silently break one already validated.
Of the 44 bugs the authors report across SQLite, DuckDB and PostgreSQL, developers had fixed 34. The authors suggest witness replay could extend to test generators beyond databases, a claim the abstract does not test.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleFramework and Benchmark for Code-Driven Agentic Testing in Web DevelopmentBin Hong, Zhenchao Zhang, Jiyuan He, Kai Zhang, Zhenya Huang
articleGrounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test GenerationMichele Tufano, James McClure, Jos\'e Cambronero, Runxiang Cheng, Sherry Y. Shi, Renyao Wei, Dorothy Chen, Franjo Ivan\v{c}i\'c, Livio Dalloro, Pat Rondon
articleAuditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle ProblemYunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni