Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem
- Source
- arxiv.org
- Author
- Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni
- Date

Context
Invalid test inputs can inflate an AI test generator's apparent success at finding bugs. The study abstract examines a workflow that treats one accepted program's output as the correct answer. If a generated input violates the task's rules or leaves behavior unspecified, differences between programs can be counted as faults without establishing a real bug.
After auditing this effect, the authors report that independently generating fresh tests outperformed repeatedly modifying earlier tests under equal budgets. They also found no robust benefit from detailed feedback in their tested comparisons. Checking several accepted implementations helped expose questionable inputs, but human review still found valid cases among inputs those programs disagreed on.
articleCorrect Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test GenerationYunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis
articleLineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided OptimizationShanglin Yang, Zhan Shi
Checking sign-in…
Loading comments…
