Intel / article

How well LLM-based test generation techniques perform with newer LLM versions?

Source: arxiv.org
Author: Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
Date: Published Jul 30, 2026

Context

A test-generation workflow can lose its advantage when the model underneath it improves. Researchers rebuilt four specialized tools with newer models and compared them with direct model prompting on 393 classes and 3,657 methods. The simpler approach could outperform the tools on the study's test-effectiveness measures at a comparable number of model requests. They also found a more economical order of work: generate tests for whole classes first, then target methods still uncovered. That strategy maintained comparable effectiveness with about 20% fewer requests. The finding supports reevaluating workflow complexity whenever the underlying model changes. Study abstract.

Key takeaways · AI-distilled

Re-running four test-generation tools (HITS, SymPrompt, TestSpark, CoverUp) on current models, a plain LLM prompt beat all of them: 17.7% more line coverage, 19.8% more branch coverage, 20.9% higher mutation score, at a comparable number of queries.
Those tools reported gains against old model versions and weak prompts. Their compilation- and execution-feedback scaffolding was compensating for model weaknesses that newer models no longer have.
Granularity drives cost: generate tests per class first, then chase only the methods still uncovered. That ordering matched the plain approach's effectiveness while making about 20% fewer LLM requests.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“In other words, stronger (newer) LLMs may obviate any advantage these techniques bring.”

“Our results show that the plain LLM approach can outperform previous state-of-the-art approaches in all test effectiveness metrics we used: line coverage (by 17.72%), branch coverage (by 19.80%) and mutation score (by 20.92%), and it does so at a comparable cost (LLM queries).”

“This strategy achieves comparable (slightly higher) effectiveness while requiring about 20% fewer LLM requests.”

Read the source arxiv.org