Intel / article

trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories

Source: arxiv.org
Author: Hadi Mohammadi
Date: Published Sep 2, 2026

Context

An can reach the expected customer-facing outcome while skipping a required check. Trajectory-judge tests whether automated evaluators notice such failures by inserting known faults into a simulated support workflow. Its comparison gives the same judging model either the request and final answer or the complete sequence of actions and observations.

Across 400 trajectories, the outcome-only judge detected about 45% of faults that left the expected outcome intact. The step-aware judge detected about 77%, while false alarms fell from 33% to zero among 100 clean trajectories. Step-aware took roughly three times as long. In this controlled setting, seeing the process substantially improved the judge’s ability to identify hidden procedural errors.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Read the source arxiv.org