Where simulation collapses the agent release cycle

>> So for those of you who build agents, you know how difficult building a new agent and maintaining uh an existing agent is. So if you want to change the agent harness, you want to change the prompt, the tools, anything around the the LLM that powers your agent, that can take a few hours and in in the best case. But as Fria mentioned, running offline evals on hand curated data can take a few days. And then the the worst part is if you run an AB test in production and monitor regressions, see if you get a statistically significant improvement of of a previous version that can take forever because customer feedback is through some kind of a feedback form and it can be sparse, it can be noisy and can take forever to know am I moving in the right direction or not. So I'm here to tell you that uh we have verified in production that yes simulations circuit this timeline short circuit this timeline from a few weeks you can go to less than a day sometimes even a few hours a few minutes and uh as I mentioned earlier uh you know with simulations it takes a few hours uh without simulation it takes a few hours to change the harness and offline evals and all of that's become super fast now with simulation the agent harness chain still takes a few hours running offline evals becomes a lot easier because you have a simulated user to fall back on. You're not waiting on hand curated data. You're not waiting on you know people to create data for you. And the best part if you want to know that am I doing well in production? Is this version going to do well? That can take just a few hours. So this has been a real gamecher for us at New Bank.