trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
Source
Hadi Mohammadi
Author
Hadi Mohammadi
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
If your evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → only reads the final reply, it misses over half the runs that reached the right answer the wrong way and falsely flags a third of good ones. The paper prices the alternative: step-level rubric judging, roughly 3x cost, far better silent-fault recall.
articleCommit-first LLM judging inherits the judge's own errorsIdil Gozel
articleInducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent EvaluationDarragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
articleGrounded Checklist Partial Credit for Agent Skill TrajectoriesSuliu Qin, Lu Yin, Xilu Wang