RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review
- Source
- arxiv.org
- Author
- Shuyu Guo, Wenxiang Hu, Yuyue Zhao, Yougang Lyu, Xiaohui Yan
- Date
Context
RubricReviewer makes a paper's criteria explicit before using them to produce a review and final assessment. It combines Scout, an that gathers external evidence, with Aligner, a trained model that uses that evidence to judge the paper against its tailored rubric. This separates evidence gathering from assessment while giving both the written review and final judgment a stated basis. The authors report more comprehensive reviews and better discrimination between submissions than prior systems in their experiments. The supported contribution is a review process organized around explicit criteria and supporting evidence. Framework abstract.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
“Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants.”
“the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage.”
“Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks.”
articleSakana's MLR benchmarks LLM peer review with planted contradictionsSakana AI
articleLitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review PlatformRuotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
articleLlm As A Judge Evaluate Ai AgentsOpenRouter editorial sitemap
Checking sign-in…
Loading comments…