Takeaways: close the sim-to-real gap, then data plus metrics is the whole game

our cards uh a bit. >> Um awesome. So um the core three takeaways from this talk right is about um where eval is today and how you can really remove a lot of bottlenecks to it. So we lied when we said earlier that there's just one thing you should take away from it. That one thing is still essential but there's a few key downstream things that you can unlock if you know you adopt it which is the first is if you generate your evaluation data in simulation rather than solely relying on production to get signal on how you know different agents are performing uh you'll be able to undercut or you'll be able to short circuit a lot of the uh bottleneck in in releasing you know versions of your agents much faster. The second is in order for any of these gains to really be unlocked uh you know you really need to close out the sim toreal gap. So you need to you know set up like offline online human review kind of metrics to really understand how sim performs visav real production data that you've seen uh so that you you you are able to kind of like trust the results of these simulations. Um and the third is again there's so much excitement around you know like auto research self-improving agents RSI etc. uh in an enterprise setting when you're building an agent, it really does come down to two things, data and metrics. If you have align metrics that are able to really catch the signals you care about and you have a reliable way of generating data that those metrics can give you signal on, it's then very easy to put together a loop of an agent that you know continuously improves itself from like feedback it receives from all of these places. Uh which you know again is like where the future of this field is heading.