Clip transcript
So for those of us building real agents, you know how hard things are. Uh you don't know if your agent is good, you don't know if your evals are good. You have no idea what's happening. And that was me a few quarters ago. Uh since the theme of uh this year's conference is self-improving agents. So we are also kind of jumping on the bandwagon. So this is our self-improvement loop. You ship your agent, you observe, you create robust evals. You spend a lot of time creating robust evals using automated prompt optimization with algorithms like Japa. And then you also do simulation. You run simulation and you pipe the simulation data and real data through your evals. And that gives you rich signal to then optimize your agent harness. And once you verify the agent harness optimization is good, you ship. And this has enabled us to do many many many AB tests for a given agent. And we have many many such agents across the company. So this has been a real gamecher for us in terms of improving our agents and the proof as I already showed you is that you know the quality of our agents is going up uh pretty dramatically. This is some data on how simulation really tracks real production data. So when you run uh simulations you want to compare is my simulation comparable to production data. So we looked at sim quality via evals. So far uh you know quality of uh with evals from sim and real data the correlation is pretty high and we had human review done where 80% of our domain expert labels confirmed that sims give us usable data not just for uh mature agents but also for green field agents. So this has been uh pretty useful for us in bringing up a new agent from scratch and also taking an existing agent and and massively improving it. And as as we as I mentioned, you know, simulation is not just about improving uh and sh also talked about this. We are derisking a lot of things that could have made to production. For instance, we caught a regression uh with simulation that could have made it to production, but simulation caught it. And at the same time, we also caught in another agent an issue uh which could have lowered our self-service rate making it less efficient. And all of this has resulted in pretty large improvements. Uh for instance, one of our agents, the TNPS, has 2xed uh thanks to simulation, robust evals, and investing in something super principled rather than, you know, throwing something at the wall and see what sticks. Um it's been pretty awesome to see simulation saving us so much time. So now the team uses snow globe and tries different ideas and they're able to short circuit launching AB test. They don't launch until they're happy with the same output. So imagine if you're running 10 AB tests a quarter for your agent or probably more if you have more data. Now you can just run a bunch of stuff through SIM and launch just one AB test which kind of shortcircuits launching the first five or six. And at the same time uh this this is often discussed in customer service circles that TNPS can come at the cost of self-service rate. Sometimes there's a trade-off but I'm happy to report that we are not compromising all self- service rate with SIM. We are also able to pretty dramatically improve SSR across the board. But in one particular case, we were able to improve it by 4%. All of this makes us a much more efficient company and makes sure that we