The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
An can produce an acceptable answer after wasting many steps repeating the same tool call. AI monitoring company Arize describes Signal as an agent that investigates execution traces, the records of an agent's actions, to find recurring failure patterns. Its announcement talk presents this as an additional evaluation method alongside fixed checks and model-based scoring.
The distinction is between scoring a requirement the team already knows to check and investigating what went wrong across many runs. Arize's product explanation says Signal reviews production traces, suggests prompt, code, configuration or evaluation changes, and leaves developers responsible for reviewing and testing them. These are the vendor's own descriptions; the sources supply no independent accuracy measurement.
- Arize runs over 100 million evals per month across its customers; the average team runs about 12 eval jobs, while the top teams run more than 3,800 different evaluators.
- Aparna Dhinakaran argues the thing being evaluated changed underneath first-gen evals: from answering a prompt in 2023, to tool calls, reasoning, and deep research in 2024, to loops on real-world data with sub-agents on long-horizon tasks — each jump producing a fundamentally dif
- Arize's own in-product agent, Alex, surfaced the failure modes classical evals miss — forgetting , not knowing when a task was done, getting stuck in loops — because every user interaction generates a new dynamic UI and thus a new trajectory.
- Her core claim: -as-a-judge applies a fixed rubric with fixed scores, which breaks when an agent takes a completely different trajectory on every run — so the answer is agent-as-a-judge, with the future being all three tiers (deterministic evals, LLM-as-a-judge, and agent-as-a
- Arize released Signal, a long-running agent-as-a-judge that reads submitted traces, discovers subtle failure patterns like repeated tool calls and inefficient trajectories, and can then open a PR with a proposed fix.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
“Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.”
Aparna Dhinakaran
“Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem.”
Aparna Dhinakaran
“What if the best way to an evaluate an agent was actually with an agent.”
Aparna Dhinakaran
“LLM as a judge just gives you a fixed rubric with these fixed scores.”
Aparna Dhinakaran
videoBetter AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish
videoWhy AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI
videoHow to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
video1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Checking sign-in…
Loading comments…