Large Language Models Threaten Double-blind Review

Removing author names may leave clues in a paper's choice of research problem. This study of review anonymity gave language models only titles and abstracts, together with five possible authors containing the true author and four alternatives. It tested whether the models could rank that supplied shortlist.

The researchers report better attribution than their human comparison and weaker performance when alternative candidates had more closely overlapping research interests. This suggests that subject matter itself can help narrow authorship. The experiment establishes a risk within small candidate pools; unrestricted attempts to name authors were inconsistent, and the study does not demonstrate reliable identification of arbitrary anonymous papers.

“Using only titles and abstracts from papers published after model training, we find that LLMs collapse anonymity more efficiently than humans, with belief concentrating onto a small subset of plausible authors drawn from pools of five domain expert candidates.”

“This vulnerability persists even when stylistic and bibliographic cues are excluded, indicating that stable patterns in problem framing and research focus function as latent conceptual signatures of authorship.”

“Together, these findings indicate that double blind review is vulnerable to automated semantic inference, necessitating a revaluation of how anonymity and fairness are maintained in an AI augmented research ecosystem.”