Clip transcript
works by first like wrapping your agent you know using a um uh like pointing the snow globe SDK at your agent without requiring any code changes and you know you can gro what are the tools that you really need in to mock in order to make the simulation really work. The second input is about how you want to drive or steer your your simulation. So you know who are the personas what are the use cases what are the data points that we're really simulating. Once you have those inputs ready, snow globe essentially runs the simulation and on the output you end up getting you know first just a data set. So you get like thousands of multi-turn conversations against your real agents. These conversations have you know tools mocked etc. have all of the bells and whistles needed to be actually you know useful for evaluating your agent already ready and then you run like judges on top of it so that on this data set you get you know per turn information about how your agent is really behaving and it then like pipes directly into you know your evaluation pipeline so that you can really get data on demand. Um this is an example of what a sim what the what what a concrete output of simulation is like right so this is as an example like one simulated persona and then the corresponding conversation or data point for that simulated persona so here for example this is totally synthetic a user named Maria Souza uh you know she wants to order a credit card this is who she is she's 34 a designer first-time credit card customer in addition to you know this basic information about this synthetic user we also will generate, you know, this essential grounding data as an example, this fake address, a fake credit card, etc. that stays consistent across your agent execution so that you know your agent can actually be evaluated against that. And then you also have things like tone, voice, how do they really interact? And then oops, and then when you actually run this persona through the conver through the agent, you end up getting, you know, this uh conversation that's consistent with the simulated intent, right? So you see for example that this user has very curt oneline messages in terms of how they interact because this is how the persona was set up or that you know when the agent wants to when the agent runs its tool when Maria enters the chat the agent is actually able to get a valid output mocked by the simulator that you know is able to identify who the user is where the address is the address can be verified etc. Um and then finally all of this is also grounded in like account data as well.