• LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Warns against a common low-cost agent evaluation pattern: LLM-simulated users plus LLM-as-judge satisfaction scores don't reliably predict whether an agent actually completed its task, which should change how teams gate agent releases.

• articleInducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent EvaluationDarragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
• articletrajectory-judge: What Outcome-Only LLM Judges Miss on Agent TrajectoriesHadi Mohammadi
• articleCommit-first LLM judging inherits the judge's own errorsIdil Gozel

Checking sign-in…

Loading comments…