← All IntelClip / AI AgentsWhy LLM-as-judge fails on agent trajectories; birth of Agent-as-judge
From The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI · ≈3:14
“What if the best way to an evaluate an agent was actually with an agent.”
“Agent as a judge is about adaptive dynamic analysis.”
“LLM as a judge just gives you a fixed rubric with these fixed scores.”
What’s in it
- Explains why classic LLM-as-judge evals fail for dynamic AI agents
- Introduces 'agent as a judge' as a new evaluation paradigm
- Details real failure modes: lost context, stuck loops, unclear completion
Clip transcript
that it would forget context. It wouldn't know when something was done. Um sometimes it would just get stuck in these loops. And the key thing here is that the classical LLM as a judge evals, that probably many of you have written in this room, just weren't for us to be able to catch all the types of failures that we were experiencing. I mean, it's just fundamentally different, right? You have a deterministic flow, and now what we have is literally every time a user interacted with Alex, it would create a new UI. That's a fundamentally different trajectory. So, this led to our really big revelation. What if the best way to an evaluate an agent was actually with an agent. Doesn't mean that all of the ways that we did evals, with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem. Agent as a judge is about adaptive dynamic analysis. LLM as a judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing, but when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.
Comments
Sign in to comment.
Loading comments…