Intel / video

SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)

Source: youtube.com
Author: AI Engineer
Date: Published Jul 29, 2026

Context

Nubank's customer-support team uses simulated conversations to screen changes before exposing customers to them. In its joint presentation with simulation provider Snowglobe, the companies describe synthetic customers paired with simulated tool responses and consistent account details. The real agent handles these conversations, and automated judges assess its behavior.

The speakers attribute a 20-fold improvement in release speed to this approach: teams can reject weak variants without waiting for every experiment to collect live feedback. They still compare simulations with real conversations and human judgments, then run production A/B tests on promising changes. The simulation supplies earlier evidence for choosing what to test with customers.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.

Key quotes

“If you generate your eval data in sim instead of waiting on production data you can ship agents 20x faster and we'll give you evidence for that.”
Aman (Nubank)

“now for multi-turn agents each data point is a trajectory with a lot of internal tool calls that all need state”
Shreya (Snowglobe)

“production traces in comparison are almost free. You don't have to pay for them. You're going to get them anyway. But the cost is that you're testing on real live users”
Shreya (Snowglobe)

“you don't know if your agent is good, you don't know if your evals are good. You have no idea what's happening. And that was me a few quarters ago.”
Aman (Nubank)

“in an enterprise setting when you're building an agent, it really does come down to two things, data and metrics.”
Shreya (Snowglobe)

Read the source www.youtube.com