← All IntelClip / AI AgentsClassic ML hygiene still applies to agents
From How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads · ≈12:22
Agents overfit to their eval data like any ML system, so held-out test sets used sparingly and refreshed from production remain the guard against generalization failure.
What’s in it
- Agents overfit to their eval data like any ML system, so held-out test sets used sparingly and refreshed from production remain the guard against generalization failure.
Clip transcript
about. Uh cool one other thing. So like all ML systems some things still are applicable which is my favorite part. So agents will not generalize very well depending on the kind of like data sets that you've trained on. It's usually a good idea to have some sort of data set to test for like edge cases like broader capabilities. Uh also a very good practice to have a test set of sorts. If some of you have worked in like traditional ML systems, it was always good to have test validation sets of that sort. Same concept applies here. Uh if you have a test set, use it sparingly and also refresh it with prod data.
Comments
Checking sign-in…
Loading comments…