← All IntelClip / AI AgentsCalibrating LLM judges by monitoring disagreement with humans
From How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads · ≈9:39
The practical mechanism for trusting an auto-rater: a standing sampling pipeline that tracks human-vs-LLM agreement over time, plus spot checks on the judge's reasoning rather than just its verdict.
What’s in it
- The practical mechanism for trusting an auto-rater: a standing sampling pipeline that tracks human-vs-LLM agreement over time, plus spot checks on the judge's reasoning rather than just its verdict.
Clip transcript
train your agent better. Okay, so now we talked a little bit more about involving cross functional teams and human agents, but what about if you're using LLM raiders, auto raiders, LMS judges. So we ended up going down that path. Also few things that we tried to do to kind of set this set a more uh comprehensive structure for ourselves is one we we monitored disagreements or in some cases teams monitored agreements. So basically if you can have a sample pipeline of sorts that is monitoring how a human raider or some expert would rate an eval versus how an LLM would rate it. You can get a sense of like how it's trending and if the agreement rates are in the ballpark that you would expect it to be. Uh the second thing was we went a little bit beyond pass and fail. So we also looked at agent traces which we'll get to later. But when we were doing pass fail eval and trying to understand how things were getting rated, we did a couple of spot checks to understand the reasoning behind those logic so we could really see what was going on. How did it come to the conclusion that something was a final pass or no? Uh and again like just reinforcing the high quality ground truth point that Daniel made earlier. So we want to give a golden set that's like super expansive. It covers a broad range of use cases and it also has very high human human agreement within your team.
Comments
Sign in to comment.
Loading comments…