Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation

An AI-generated test needs both an input that exposes a bug and the correct expected answer for that input. Improving one does not necessarily improve the other: a model can become more accurate by choosing easy cases that reveal fewer faults.

This study trained a Qwen model using separate feedback for test correctness, potentially revealing inputs and faults actually detected by correct tests. Effective fault detection rose from 12.23% to 14.15% on the study's fixed bank of faulty programs. But a comparison without fault-detection rewards achieved slightly higher effective detection while selecting less revealing inputs. The result shows why useful test generation needs those measurements kept separate; a higher correctness score alone does not explain what improved.

“These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict.”

“These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.”