But you can ask, "Okay, but why do we need it? Like we already have public benchmarks." The challenge with public benchmarks is that usually they are focused on a very specific domains. For example, SweepBench is focused on like uh fixing GitHub issues, TerminalBench will focus on agent running in terminal, and Co- Bench will focus on computer use agent. In your case, you want your benchmark to be focused on your company's domain both from perspective of use cases and in terms of tooling that your agent has, whether it follows the policies that your company uses, and whether you get full production environment. Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship. And a lot of the time public benchmarks they're specifically focused on pass rate. Every time you see new model release, you see performance like pass rate on different benchmarks. Which makes sense because it tells us like about the frontier of how good is the like new model. But when you release agent to production, you also care about more metrics. You care about cost for solving the task. You care about latency. You care about number of retries. And by running evaluation offline with in simulations, you can effectively compare apples to apples and and iterate on agent. In this case, you can test the full stack of your agent. You're not just checking, okay, is model one is one model performs better than the other model. You check even like thinking talk like thinking level. You can change the prompt. You can tune the full hardness and skills tools available to agent because in your production, you don't care about the model, you care about the full system. And here you can configure and test the full system while keeping environment and evaluators the same between runs.