Sakana AI benchmarks LLM peer review with planted contradictions
- Source
- x.com
- Date
Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review Peer review needs support, not substitutes. Accepted at TMLR: our new paper on using AI to help reviewers catch errors in research papers. https://t.co/KqDLUPkwib As research submissions grow, so does the workload for the experts who evaluate them. AI review systems are emerging as a potential solution, but much of their development and evaluation focuses on how closely they imitate human reviews. In our work “Beyond Imitation: A Framework and Benchmark for LLM Assisted Peer Review”, we explore how AI can support the review process without losing the human touch. We focus on one demanding but essential task: catching errors in research papers. We build an automated pipeline that deliberately introduces contradictions into papers by adding statements that conflict with information elsewhere in the manuscript. These planted errors give us clear targets for testing whether AI reviewers can spot and explain what is wrong. Not all errors carry the same weight. Some undermine a paper’s central findings; others affect smaller details. We map the connections between each paper’s claims, methods, and evidence in…
It shows how to evaluate reviewers on catching errors rather than imitating human reviews, using planted contradictions with severity weighting. The method transfers to evaluating any LLM verification .
- The comes from an automated pipeline that inserts statements contradicting information elsewhere in the same manuscript, so every test paper carries known errors that an AI reviewer must both spot and explain.
- Sakana does not score all planted errors equally: it maps how each paper's claims, methods and evidence connect in a knowledge graph to estimate severity, since some contradictions undermine central findings and others touch only smaller details.
- Its Multi-Layered Review system follows the Three-Pass Approach to reading papers: first outline the main ideas, then examine details and potential weaknesses, then combine the observations into a review. The stated principle is to understand a paper before judging it.
- In Sakana's own evaluations, Multi-Layered Review detected more errors than the other review systems tested, including on papers withdrawn because of real mistakes, while its overall quality assessments stayed broadly consistent with human judgments.
- The AI feedback emphasized different aspects of the work than human reviews did, which the authors present as a complementary perspective. Their stated aim is support for reviewers, with human expertise and judgment kept at the center.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleLitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review PlatformRuotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi- articleRubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer ReviewShuyu Guo, Wenxiang Hu, Yuyue Zhao, Yougang Lyu, Xiaohui Yan
Checking sign-in…
Loading comments…