← All IntelClip / AI AgentsSignal: an agent that finds and fixes trace-level failures
From The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI · ≈4:42
“Signal's actually a long-running agent that can read traces sent in, discover patterns of issues.”
“it was calling the same tool for repeatedly long time, the trajectory was inefficient.”
“actually what this does is because it has all that analysis, it can go put up a PR and put up a fix.”
What’s in it
- Introduces Signal, a long-running agent that hunts subtle eval failures
- Explains how it catches tool-call loops and inefficient trajectories
- Shows how it auto-opens a PR to fix issues it finds
Clip transcript
we've released agent as a judge to help our teams on their eval journey. We've released signal. Signal's actually a long-running agent that can read traces sent in, discover patterns of issues. Um, it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics. It's helped us figure out very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool for repeatedly long time, the trajectory was inefficient. And actually what this does is because it has all that analysis, it can go put up a PR and put up a fix. So, if you want to learn more,
Comments
Sign in to comment.
Loading comments…