← All IntelClip / AI AgentsHold out a benchmark validation set like classic ML
From From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI · ≈16:53
Applies the 80/20 train/validation split to agent benchmarks so config tuning doesn't silently overfit the eval set.
What’s in it
- Applies the 80/20 train/validation split to agent benchmarks so config tuning doesn't silently overfit the eval set.
Clip transcript
All right. >> A great talk. I have a question about structuring the benchmark. How many examples you should include ideally? And do you split them into, uh, like a train test split? If so, like how do you recommend structuring examples, um, across those two splits? >> Great question. So, uh, the question was basically how can we structure to for training and validation? And just as previous speakers uh shared, basically they also had a pattern of like having train validation split. This is very close to traditional machine learning where we want to have standalone data set that agent didn't see. But basically where we can verify agent config. I think the classic approach applies when like you want to have 80/20%. Always depends on the use case. But uh you do want to have a standout data set that agent didn't see through the
Comments
Checking sign-in…
Loading comments…