Intel / article

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

Source: arxiv.org
Author: Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
Date: Published Aug 17, 2026

Context

RubricForge teaches an AI evaluator what successful work looks like by improving its written scoring criteria against examples with known outcomes. It then freezes those criteria and judges new action records without accessing the task environment. The learned artifact is readable text, rather than changed model weights.

Using one fixed seven-billion-parameter model as both agent and judge, the researchers report a false-pass rate of 11.5% versus 17.3% for a generic judge on tau-bench, a difference the abstract ties to three additional over-credit catches and no reversals. Its overall agreement advantage was not statistically significant. The reported benefit is therefore narrow: slightly fewer unsuccessful runs received passing judgments in that .

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source arxiv.org