Can we trust LLMs to judge whether a research idea is novel?
Source
huggingface.co
Date
Why it matters
Tests whether an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → novelty judge can be trusted, with released data and code. Anyone building automated research or LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition → pipelines gets a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → to validate judges before relying on them.
Key takeaways · AI-distilled
In Ai2's tests, rewording one prompt changed GPT-5.4's accuracy at picking the more novel of a human and an AI-generated idea from 93.5% to 40.9%, below chance, and flipped its verdict on more than half of the same pairs.
The same prompt change moved accuracy by only about 7 points on human-only pairs, so Ai2 warns that a novelty judge validated on human papers can behave very differently on the AI-generated ideas it is meant to screen.
A plain Claude Opus 4.6 prompt with majority voting over three calls scored 0.854 macro-F1 for $2.59 in total, beating Idea Novelty Checker (0.642 at $30.66) and Sakana's AI Scientist evaluator (0.772 at $82.05).
More compute did not help: an agentic GPT-5.6 Sol judge with extra-high reasoning and live arXiv search scored 0.78 macro-F1, 11 points below Claude Opus 4.6, at roughly 10x the cost per decision.
Ai2 labeled ideas using ICLR 2026 review passages that explicitly discussed originality, not acceptance alone, because accepted papers can lack novelty and rejected papers can still be original.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
articleThe Limits of Automatic Evaluation of Creativity in Large Language ModelsAlessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
articleVibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting AgentsToshiaki Koike-Akino, Vlad Blaykhman, Ye Wang, Jing Liu, Gene V. Vinokur