← All IntelClip / AI Agents
How the eval target has changed since 2023
From The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI · ≈2:00
“So, we added evals, they catch all the failures, right?”
“Now, what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks.”
“Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem.”
“What that meant is that as these systems got more complex, so did the way that they actually fail.”
What’s in it
- Explains why old-school evals miss failures in today's AI agents
- Traces AI's leap from single prompts to multi-agent, long-horizon loops
- Argues each capability jump created a wholly new failure mode, not just harder ones
Clip transcript
So, we added evals, they catch all the failures, right? Here's the problem. When we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us. In 2023, it was about just answering a prompt. In 2024, we started to see all the frontier models. They've added tool calls, they've added reasoning, they've added deep research. Now, what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks. Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem. What that meant is that as these systems got more complex, so did the way that they actually fail. We're really lucky cuz we have our own
Recommended reads
Comments
Sign in to comment.
Loading comments…