Empty Intersection: Provenance Coverage Rose to 98% and Neither Verification Decision Moved
Source
Dong Hyeon Jeon
Author
Dong Hyeon Jeon
Date
Key takeaways · AI-distilled
On a frozen production snapshot of 194,620 rows, the paper tests two provenance defenses, a grade on every row and a single write ingress that enforces it, against the two verification decisions the snapshot supports. Neither gave either decision admissible input.
Widening the grade vocabulary raised classified coverage from 36.1% to 98.4%, and the single ingress refused 3,070 ungraded writes, yet the rows these fixes repaired never overlapped with the 32 rows the two decisions actually read.
The author's diagnosis: both prescriptions are stated over the whole population and never name which rows a decision will read, so coverage metrics can climb to 98% while the verdicts that matter stay blocked.
Filtering the verification queries by provenance grade turned both decisions from pass to undetermined. A companion paper had found the routines were deciding outcomes from values the system itself had written.
Why it matters
Structural provenance fixes for a production verification system raised classified coverage from 36% to 98% and blocked thousands of ungraded writes, yet changed neither of the two real verification decisions they were meant to fix.
articleBeyond "Made with AI": Visualizing Provenance Density to Mitigate the Transparency PenaltyQing Zhang, Yifei Huang, Juyoung Lee, Thad Starner, Jun Rekimoto
articleSilent Success: A Release Gate That Passed on Checks It Never Ran, and Eight MoreDong Hyeon Jeon
articleThe reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agentsAchint Mehta