Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms
Source
Serhii Yemets, Marek Horv\'ath
Author
Serhii Yemets, Marek Horv\'ath
Date
Key takeaways · AI-distilled
Fine-tuned CodeBERT reaches 92.6% Top-1 authorship accuracy for 10 authors on Google Code Jam (70.7% at 1000 authors) — then collapses to 0.2% and 0.06% Top-1, at or below chance, on real university coursework.
A companion multi-model benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → shows the collapse is not a CodeBERT quirk: the gap persists across every evaluated model family, pointing at dataset and task properties rather than architecture.
The actionable warning: competition-based benchmarks wildly overestimate practical authorship attribution — any academic-integrity or code-provenance deployment must be validated on the actual target coursework context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → first.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Fine-tuned CodeBERT models that hit 92.6% Top-1 accuracy on Google Code Jam authorship attribution drop to 0.2% and 0.06% on real university coursework, and the gap holds across model families — a warning for anyone deploying ML-based authorship detection for academic integrity or code-provenance claims.
Key quotes
“On multi-round GCJ data, CodeBERT reaches 92.6% Top-1 accuracy for 10 authors and retains 70.7% Top-1 (88.2% Top-10) for 1000 authors.”
“On the examined coursework datasets, the same pipeline performs at or below the corresponding chance baselines: 0.2% Top-1 on a closed-assignment dataset of 690 authors and 0.06% Top-1 on open-ended assignments evaluated over 812 authors.”
“We analyze dataset and task properties that plausibly explain this gap and argue that GCJ-based benchmarks overestimate the practical applicability of authorship attribution in educational settings unless they are validated on the target coursework context.”
articleThe Leaderboard Illusionarxiv.org
articleKeep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference AttacksDongdong Zhao, Jian Chen, Guancheng Lin, Jianwen Xiang, Jacky Wai Keung, Xiao Yu