← All IntelClip / AI AgentsPatterns, not isolated runs
From How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads · ≈14:46
Names the most common eval trap — patching the prompt after one bad trace in a non-deterministic system — and the fix: multiple golden-set examples per pattern and a failure rate per pattern.
What’s in it
- Names the most common eval trap — patching the prompt after one bad trace in a non-deterministic system — and the fix: multiple golden-set examples per pattern and a failure rate per pattern.
Clip transcript
and figure out um the trade-offs here and then um you should also this is a very important point so you should focus on patterns rather than isolated runs so a tempting thing is to hyperfixate on very small examples from the model right so you might have uh one run that you do with your agent and then you find uh it fails on this case and you might think to yourself, okay, well, I should update the prompt based on my eval and uh the the trace of the agent, right? But um if you do this, this is kind of a trap because as I said before, these are non-deterministic systems. So what's more important is that you actually rely on patterns. So in your golden set, it's important to have multiple examples that can cover these kind of uh patterns that you might want to see. And you basically want to look at the entire picture of how often is it failing on that pattern, not that specific individual example. Um, and yeah, it's important here also to invest in online evals uh and and making sure that your data is matching the real world representation.
Comments
Sign in to comment.
Loading comments…