VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories

A test can pass without checking the behavior it was written to protect. VibeCheck gives a concrete example: a generated test checked an error only inside an exception handler. If the code failed to throw the required error, the handler never ran and the test could still pass.

The study examined generated tests across 15 student-developed repositories and found recurring weak assertions and missing cases. Its quality assessments came entirely from other AI agents, without human validation, which limits the reported tool rankings. The example nevertheless explains the review question that execution success misses: would the test actually fail if the required behavior disappeared?

“Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage.”

“Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow.”