Intel / article

Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms

Source: arxiv.org
Author: Serhii Yemets, Marek Horv\'ath
Date: Published Aug 4, 2026

Context

Strong results identifying competitive programmers did not carry over to the examined student coursework. A study using CodeBERT trained a model to identify a program’s author from its code. It reported 70.7% first-choice accuracy among 1,000 authors in a multi-round competition dataset, but only 0.2% among 690 authors completing prescribed coursework and 0.06% among 812 authors completing open-ended assignments.

The coursework results were near or below their respective random-choice baselines. Because the datasets differ in code volume and task characteristics as well as population, the findings support validating attribution on the actual educational setting before relying on competition accuracy.

Key takeaways · AI-distilled

Fine-tuned CodeBERT reaches 92.6% Top-1 authorship accuracy for 10 authors on Google Code Jam (70.7% at 1000 authors) — then collapses to 0.2% and 0.06% Top-1, at or below chance, on real university coursework.
A companion multi-model shows the collapse is not a CodeBERT quirk: the gap persists across every evaluated model family, pointing at dataset and task properties rather than architecture.
The actionable warning: competition-based benchmarks wildly overestimate practical authorship attribution — any academic-integrity or code-provenance deployment must be validated on the actual target coursework first.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“On multi-round GCJ data, CodeBERT reaches 92.6% Top-1 accuracy for 10 authors and retains 70.7% Top-1 (88.2% Top-10) for 1000 authors.”

“On the examined coursework datasets, the same pipeline performs at or below the corresponding chance baselines: 0.2% Top-1 on a closed-assignment dataset of 690 authors and 0.06% Top-1 on open-ended assignments evaluated over 812 authors.”

“We analyze dataset and task properties that plausibly explain this gap and argue that GCJ-based benchmarks overestimate the practical applicability of authorship attribution in educational settings unless they are validated on the target coursework context.”

Read the source arxiv.org