Vibeleaderboard
← All Intel
Intel / article

RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

Source
arxiv.org
Author
Shuyu Guo, Wenxiang Hu, Yuyue Zhao, Yougang Lyu, Xiaohui Yan
Date

Context

RubricReviewer makes a paper's criteria explicit before using them to produce a review and final assessment. It combines Scout, an that gathers external evidence, with Aligner, a trained model that uses that evidence to judge the paper against its tailored rubric. This separates evidence gathering from assessment while giving both the written review and final judgment a stated basis. The authors report more comprehensive reviews and better discrimination between submissions than prior systems in their experiments. The supported contribution is a review process organized around explicit criteria and supporting evidence. Framework abstract.

Terms in this piece · Glossary
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Key quotes

“Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants.”

“the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage.”

“Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks.”

Recommended reads
Comments

Checking sign-in…

Loading comments…