Intel / article

Sakana's MLR benchmarks LLM peer review with planted contradictions

Source: arxiv.org
Author: Sakana AI
Date: Published Oct 9, 2026

Context

A Sakana AI paper published in Transactions on Machine Learning Research argues that AI peer-review tools should be tested on catching errors, not only on matching human review scores (paper). The authors planted contradictions in openly licensed AI conference papers, giving each test a known answer. An mapped each paper's claims and evidence into a graph, and a contradiction counted as more severe the closer it sat to a main claim.

The paper's Multi-Layered Review system pairs an appendix and an optional literature-review agent with a Review Agent that reads the main text in three passes. The contradiction and withdrawn-paper tests used only the Review Agent. With four reviews per paper, it caught more than 70% of the most severe contradictions, about four times the next strongest baseline. Part of that edge comes from using Claude rather than the baselines' OpenAI models, though the design added a further large gain.

Planted errors are an imperfect stand-in for real ones. In an audit of 50, human raters judged 34% to lack coherence with adjacent sentences, and every system did worse on real errors in withdrawn papers.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.

Read the source arxiv.org