Diagnosing Generalization Boundaries in AI-Based Vulnerability Detection via Calibrated Embedding Distances
Source
arxiv.org
Author
Konstantin Ievtushenko, Ralf L\"ammel, Jan J\"urjens
Date
Why it matters
benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → scores for vulnerability detectors hide steep drops under cross-project shift. Embedding distance can flag inputs where the model is unreliable.
Key takeaways · AI-distilled
Cross-project shift caused the sharpest accuracy loss, even though temporal shift moved samples farthest in embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → space, so geometric distance alone does not rank which shift hurts most.
False positives on patched code tracked proximity to vulnerable training examples, rising from 9.4% in the lowest-risk decile to 58.8% in the highest.
The method uses whitened intermediate-layer embeddings with local nearest-neighbour scoring on strictly paired vulnerable and patched code, and needs no auxiliary model.
Terms in this piece · Glossary
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleKeep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference AttacksDongdong Zhao, Jian Chen, Guancheng Lin, Jianwen Xiang, Jacky Wai Keung, Xiao Yu
articleBeyond "Made with AI": Visualizing Provenance Density to Mitigate the Transparency PenaltyQing Zhang, Yifei Huang, Juyoung Lee, Thad Starner, Jun Rekimoto
articleAutomated Vulnerability Injection in Smart Contracts Using Large Language ModelsLuca Migliaccio, Roberto Natella, Naghmeh Ivaki, Nuno Laranjeiro, Marco Vieira