← All IntelClip / AI AgentsEvals encode failures you've already seen
From From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize · ≈19:23
Positions LLM-as-judge evals as a ratchet against known regressions — find a recurring failure, create an evaluator for it, and it runs continuously at full dataset scale.
What’s in it
- Positions LLM-as-judge evals as a ratchet against known regressions — find a recurring failure, create an evaluator for it, and it runs continuously at full dataset scale.
Clip transcript
it's using data from the evals that were layered on um in addition to all the raw data that it has there Um it but it tends to be like you build an eval for a failure you've seen before a lot of times. So I have these prompt injection things that I'm trying to catch or something or um or or a failure in the way it's responded maybe to to something before. So they they tend to be this like you know u at least the LM as a judge is tends to be like this this thing you um preset up and then you can actually create evaluators for failure. Say you find this failure that's pretty common and happening all the time. I can create an eval so I can catch it next time. I just you think of it as like almost a an AI um assessment that's always running. Uh the other note is the element as a judge can run really at scale. Well, every you know I have customers who who lay you know layer element as a judge across um their full data set uh where where this tends to be like you know uh more periodic on a lot of data. So cool. Thank you.
Comments
Sign in to comment.
Loading comments…