Silent Success: A Release Gate That Passed on Checks It Never Ran, and Eight More
Source
Dong Hyeon Jeon
Author
Dong Hyeon Jeon
Date
Key takeaways · AI-distilled
A blocking production release gate reported PASS for two weeks while one subgate ran neither of its two checks and another ran six of eight, because checks that failed to run were dropped before counting violations.
The fix was a third outcome, unable to determine, alongside pass and violate, which turned the silent passes into failures and surfaced the shortfall in the exit status.
The author documents eight more cases of the same pattern, including a cache TTL declared but never applied on the write path and an inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → server crediting free cache slots for releases that freed nothing.
Across the nine cases, seven could be caught with a single query, command or comparison, which is the falsifiable claim the paper asks to be judged on.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
A case study documents nine real release gates and caches that reported success while silently skipping checks (one stayed green for two weeks); adding an explicit 'unable to determine' verdict alongside pass/fail exposed the hidden failures.
articleWhen Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic SystemsWenji Bai, Muhammad Waseem, Zeeshan Rasheed, Jaakko Peltonen, Pekka Abrahamsson
articleEmpty Intersection: Provenance Coverage Rose to 98% and Neither Verification Decision MovedDong Hyeon Jeon
videoComputer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma LabsAI Engineer