GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
- Source
- arxiv.org
- Author
- Umesh Bodhwani, Thanh Tran, Kai Wei
- Date
Context
An conversation can sound helpful even when the customer's task fails. In GAUGE, researchers at Amazon compared ratings of simulated service conversations with checks of task completion. Among conversations a human panel rated satisfactory, 57.5% failed, almost identical to the sampled conversations' overall 57.3% failure rate. Satisfaction therefore added little information about success in this sample. The paper reports the comparison separately from its tests of automated judges.
Judges given the task, tool calls and operating rules were more informative than satisfaction alone. This makes the target consequential: matching people's impression of a conversation is insufficient evidence that an agent completed the requested work.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
articleWhen Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to GuessSalma Roshdy Aly, Hussein Assaf, Ziad Kobti
articleInducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent EvaluationDarragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
articletrajectory-judge: What Outcome-Only LLM Judges Miss on Agent TrajectoriesHadi Mohammadi
Checking sign-in…
Loading comments…