Can we trust LLMs to judge whether a research idea is novel?

Systems that generate research ideas often rely on a second language model, a "novelty judge", to decide whether an idea is genuinely new. A study from Ai2 finds those judges unreliable, especially on the AI-generated ideas they are meant to screen. Across six judges from OpenAI's GPT-5 line and Anthropic's Claude line, small changes to the setup, such as a few words of prompt, produced very different verdicts on the same ideas, according to Ai2's post on the paper, Old Ideas, Novel Problems.

To test the judges, Ai2 built two sets of ideas from ICLR 2026 peer reviews. Instead of treating acceptance as a sign of originality, it mined the reviews for passages where reviewers discussed novelty. That gave 154 ideas from highly rated accepted papers whose reviewers praised their originality and 145 from low-rated rejected papers whose reviewers questioned it. A second set paired the 154 strong human ideas with 154 ideas from a basic language model given only a research area and an instruction to propose something novel. Ai2 assumed the generated ideas were less novel; this is an assumption, not a verified label for each idea, though an expert who reviewed 30 pairs preferred the human idea every time.

The clearest failure came from prompt wording. When GPT-5.4 was told that reviewers had found exactly one idea in each human-versus-generated pair to be novel, it picked the right one 93.5% of the time. Asked simply which idea was novel, its accuracy fell to 40.9%, below chance, and it changed its verdict on more than half the pairs. The same change moved accuracy by only about seven points on the human-only set, so a judge that looks stable on human research can behave differently on generated ideas. No prompt was best for every model. When the judges compared ideas from two AI generators, roughly a quarter to a third of verdicts flipped across the tested prompt changes, a disagreement that shows how much the judge's prompt can change a comparison between idea-generating systems.

More effort did not fix this. A GPT-5.6 Sol judge with extra reasoning and live arXiv search scored 0.78 macro-F1, a measure of how well it sorts novel from non-novel ideas, against 0.89 for a Claude Opus 4.6 judge, at roughly ten times the cost per decision. Two purpose-built tools, Idea Novelty Checker and the novelty evaluator in Sakana AI's AI Scientist, scored 0.642 and 0.772 while costing $30.66 and $82.05, against 0.854 and $2.59 for a simple prompted Claude Opus 4.6 baseline. Ai2 concludes that judging novelty with language models remains a major open problem, and that judges should be tested on the kind of ideas they will actually evaluate and across prompts and models. It has released the benchmark, data, code and its review-mining pipeline for that purpose.