Vibeleaderboard
← All Intel
Intel / article

Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem

Source
arxiv.org
Author
Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni
Date

Context

Invalid test inputs can inflate an AI test generator's apparent success at finding bugs. The study abstract examines a workflow that treats one accepted program's output as the correct answer. If a generated input violates the task's rules or leaves behavior unspecified, differences between programs can be counted as faults without establishing a real bug.

After auditing this effect, the authors report that independently generating fresh tests outperformed repeatedly modifying earlier tests under equal budgets. They also found no robust benefit from detailed feedback in their tested comparisons. Checking several accepted implementations helped expose questionable inputs, but human review still found valid cases among inputs those programs disagreed on.

More from Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni
Recommended reads
Comments

Checking sign-in…

Loading comments…