So now that we've kind of established why we use judges, uh we want to go through some of the general heristics and kind of principles we think about when we're designing good judges. Um some of the things that we think about at data. So you know I think the first important uh consideration to make is that judges are agents too. Um, you know, so as environments get really complex, oftent times we a consideration we kind of have is like, hey, we have to make sure the harness can kind of scale and and kind of match up with whatever environment uh you kind of have. Maybe that means introducing a bunch of new tools and making sure your harness can support those tools really well. The agent has clear observability over what's happening in the environment. Um, but I think like we said, the way the judge determines correctness is that it oftent times has to look at the state of the environment itself as well. So a lot of the harness that you've designed for the agent might also be reused uh for the judge as well. Um, I think the best way to illustrate this is the example we have here. Let's say you've defined a task where, you know, there's some deployment failure with the software engineering task of some platform you're deploying and the agent's task is to like sift through the CI/CD logs on GitHub, look through the cloud cloudatch logs, figure out whatever happened, uh, kind of apply the changes you care about to the codebase and then open a PR and and kind of kick off a redeploy there once the PR is merged. Um for a lot of for a lot of that if the judge actually wants to verify uh whether or not this is correct besides just like looking at the tool calls agent mate which are usually not very reliable it actually has to also check the GitHub logs it might check the AWS logs or the GitHub logs after the deployment happened to make sure oh are things actually working properly. So it's really important that the judge has access to the environment in the same way uh with some important safeguards of course. One is that we don't want the judge to make an accidental mutation in some way to the environment after the agent is done. So you want to be very careful about that. Maybe that means enforcing readon permissions for a lot of this information. It can't actually kick off a deployment or anything like that. So those are things to be careful about. But I think this is really really important especially where there's a lot of open-ended approaches and the only way we can really verify correctness is to actually look at the state itself. Uh you the answer isn't obvious of whether or not the agent completed the task just from looking at the trajectory. So I think that's one example where this approach is really really important. Um I think uh the other thing to be notable of is you know as these environments get more complex the agent trajectories get longer and longer and part of the reason we also need the judge to be an agent is that you can't just use this really basic approach of taking the trajectory and stuffing it in the context window of the judge and kind of have it be a basic LM call. Uh these trajectories can get really really long and really really complex. So we need to do a lot more thoughtful uh processing of the trajectory in some meaningful way. So you know that might mean we put into some database. We use sub agents to actually enrich certain information. Maybe we parse out specific phases that the agent was actually in. Maybe the beginning part was it going through logs. The second part was actually writing code. The third part was actually it checking what happened after that. These are all different things we want we want to do. And in that sense we need to make the trajectory itself queryable. So that might mean enriching of information like I just said or some other metadata we can kind of look at at certain steps. is really important so the agent can find critical steps um you know like failure points and and verify whether or not those are actually failures. Making that uh kind of usable for the agent is really really important.