Vibeleaderboard
← All Intel
Intel / article

Language Models Make New Research Sound Old

Source
arxiv.org
Author
Guy Zyskind
Date

Context

Researchers increasingly ask a whether a question they want to work on has already been studied, and this single-author preprint finds that the models' ordinary answers make open questions sound partly answered, even when the same model, asked separately, says the question is open. Guy Zyskind of the University of Miami tested six frontier systems from OpenAI and Anthropic on 40 research questions taken from papers accepted at the 2026 Conference on Language Modeling (COLM). The papers appeared after the models were trained, their search was restricted to work available before the submission deadline, and peer review supplied the evidence that the questions were new. Asked for a plain yes or no, GPT-6-Astra judged 98% of the questions unstudied, yet its normal answers said so plainly only 8% of the time. Every system showed a gap of 30 to 90 percentage points between its direct judgment and its ordinary answer.

The gap comes mostly from how the answers describe related work. A typical answer concedes that the question is open, then describes the nearest paper as having done most of it. An independent grader, itself a model, compared each cited paper with the question's core parts (task or setting, method, outcome and study design). On the accepted-paper questions, 34 of 43 claims that a paper nearly answered the question were wrong, and the answering models rejected 31 of those 34 when asked to check the papers separately. On constructed test questions, swapping the closest paper for one that shared only the topic cut hedged answers across the six systems from 61% to 33%. The failure runs one way: when a question really had been studied and the answering paper was supplied, the models said so plainly in 88 to 100% of answers. Across 22 models given identical papers, the more capable ones judged study status more accurately and also had larger reporting gaps.

The fix the paper tests builds the answer from the model's own work. A template states the model's separate judgment first, explains what the closest paper leaves unanswered using the model's own check of that paper, and lists other papers as related work. That raised Astra's plain reporting of open questions from 8% to 95%, and when all six systems read the same paper lists, plain reporting rose from 52% to 92%. Telling the model to lead with its judgment did less, raising it only from 52% to 63%, because the later description of prior work still made the question sound partly answered. Phrasing the request as "Is this research question novel?" produced the largest gaps of the five wordings the paper tried on two of the systems. The work complements Ai2's study of AI novelty judges, which found that changing a few words in a prompt could shift a model's accuracy at picking the more novel of two ideas by more than 50 percentage points. Zyskind's paper asks a different question: whether a model reports the judgment it has already reached.

The evidence has clear limits. The paper has not been peer reviewed, most questions concern machine learning, and models rather than people labelled the answers and graded the cited papers. Its test of how researchers would react to hedged advice used AI models as stand-in readers, so the effect on real research decisions is untested. Within those limits, the study supports treating a model's direct verdict, plus a separate check of the closest paper it cites, as more reliable than its prose summary of the literature.

Key takeaways · AI-distilled
  • On 40 open questions from newly accepted papers, GPT-6-Astra correctly judged 98% to be unstudied, but stated that plainly in only 8% of ordinary answers, the paper reports. All six frontier systems tested showed 30-90 point gaps between judgment and report.
  • Across 22 models given identical papers, more capable models judged novelty more accurately but also showed larger reporting gaps. The failure is one-sided: with the answering papers supplied, models identified already-studied questions in 88-100% of ordinary answers.
  • The proposed fix builds each answer from the model's own judgment plus separate checks of every cited paper. It raised Astra's plain reporting of open status from 8% to 95% and improved all six systems.
  • Combining several models' written ratings with a rule learned from graded examples predicted reviewers' novelty grades within the range reported for Think-Probe-Respond, without access to weights or activations, so it also works on closed-weight models.
Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Recommended reads
Comments

Checking sign-in…

Loading comments…