Memorization Diagnostics for Code LLMs Should be Scale-Aware
Source
arxiv.org
Author
Prateek Kumar Rajput, Abdoul Aziz Bonkoungou, Alberick Euraste Djir\'e, Xunzhu Tang, Yewei Song, Iyiola Emmanuel Olatunji, El Hacen Diallo, Jacques Klein, Tegawend\'e F. Bissyand\'e
Date
Why it matters
If you use contamination probes to judge whether a code model memorized a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →, those probes likely fail on exactly the large models you ship — the methodology, not the model, is what breaks.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleThe Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory AgentsJundong Hu, Shekar Ramachandran
articleMeasure, Don't Optimize: Forecasting Recovery in LLM UnlearningZirui Song, Huaxing Liu, Xiang Wang, Shuai Li, Xinye Li, Lang Gao, Jinghui Zhang, Zheng Lu, Fengxian Ji, Xiaojun Chang, Xiuying Chen
blogMitigating memorization in LLMsblog.janestreet.com