Metrics are solved, eval data is the bottleneck

implementing uh you know this framework at scale in new Uh, awesome. So, eval there this there's like a whole talk track dedicated at this conference to eval. Evals are absolutely critical for building good agents and evals are really only about two things, right? There's metrics and there's data. uh metrics again I hope you attended many of the amazing talks yesterday on the evals track but metrics are really while they're challenging we have a playbook for how to build metrics that are really well aligned with you know the rubrics that we care about right which is you essentially use um you know LM as a judge style uh you know classifiers and you align it with human judgment and and getting human data and you can iteratively build on it using you know auto optimization and autoprompt tuning uh techniques the thing that's a bottleneck And that still remains very challenging and unsolved is what is the data that you're actually computing these metrics on and that process is very timeconuming and very expensive specifically so for agents. So once again people have been talking about you know if if you're around like machine learning era circa you know 2018 it would be like ML work is 85% data work right so data has always been challenging but with agents the level of sophistication that data requires is just so much more expensive. So here's examples you know of what structured ML data looked like uh and what even early era of AI uh data for you know chat bots or single turn QA which was you know much more manageable and tractable and you could still think of it as these structured rows. Uh but now for multi-turn agents each data point is a trajectory with a lot of internal tool calls that all need state etc. And that becomes very even one data point is expensive to generate and annotate right but scaling it up on a sample set becomes even harder. Um so how teams primarily get eval data today is two methods. Uh one is either manual or the second is production traces and both of those are painful but for different reasons right manual authoring I mean this one's obvious it is so timeconuming especially the kind of data that I was showing you on the previous slide. You have to hand plan you know state updates. You have to hand plan the trajectory that you want to take this uh you know agent data point in and synthetic uh the the state for these data points has to stay consistent across all tool calls etc that you make right production traces in comparison are almost free. You don't have to pay for them. You're going to get them anyway. But the cost is that you're testing on real live users every time you're testing it. So for example, if you want to run, you know, many parallel experiments at scale, which is often something you want to do in a machine learning setting is it becomes very hard to do that in prod.