How well LLM-based test generation techniques perform with newer LLM versions?
Source
Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
Author
Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
Date
Key takeaways · AI-distilled
Re-running four LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → test-generation tools (HITS, SymPrompt, TestSpark, CoverUp) on current models, a plain LLM prompt beat all of them: 17.7% more line coverage, 19.8% more branch coverage, 20.9% higher mutation score, at a comparable number of queries.
Those tools reported gains against old model versions and weak prompts. Their compilation- and execution-feedback scaffolding was compensating for model weaknesses that newer models no longer have.
Granularity drives cost: generate tests per class first, then chase only the methods still uncovered. That ordering matched the plain approach's effectiveness while making about 20% fewer LLM requests.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
Scaffolding tuned to older models can become dead weight — or a regression — when the model underneath changes.
Key quotes
“In other words, stronger (newer) LLMs may obviate any advantage these techniques bring.”
“Our results show that the plain LLM approach can outperform previous state-of-the-art approaches in all test effectiveness metrics we used: line coverage (by 17.72%), branch coverage (by 19.80%) and mutation score (by 20.92%), and it does so at a comparable cost (LLM queries).”
“This strategy achieves comparable (slightly higher) effectiveness while requiring about 20% fewer LLM requests.”
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleTangent: An Empirical Study of Testing Practices for LLM-Based Agent ApplicationsRangeet Pan, Tyler Stennett, Divya Sankar, Bridget McGinn, Alessandro Orso, Raju Pavuluri, Saurabh Sinha, Maja Vukovic
articleScaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model ParametersCharlie Snell et al.