Clip transcript
we need to also increase the complexity of our environment. So, with synthetic environments, we have a very similar setup, but we offload a lot of the environment state um outside of the training stack. So, you still have the same orchestrator um from before, but um the task back is maybe a little bit more complicated, and the environment state is uh living outside of the training stack. So, the task back might now include things like tool cost specs or like maybe an initial state for your environment, like a file system. And the orchestrator is now in charge of running many turns in a in series, where maybe first they ask the model for how it wants to respond, and then if the model wants to call some tools, it'll then call the sandbox to actually like modify the environment state or read the environment state, and then return those results back to the model. After all that is said and done, you get a full task trace out of this, um, and that task trace is then sent to a grader for grading. And very similar to what we had before, you'll be able to take the graded chats, you'll be able to then use them to do a weight update. The main thing to highlight here is that this orchestrator and sandbox setup is replayable, which is basically just saying that for any specific prompt, you can always like roll back to the initial state and like rerun it. Um, you can do that in parallel or you can do that in series. But, the reason that's important is because the main sort of method that we use for reinforcement learning today is GRPO, and that involves comparing many rollouts for the same prompt, and then comparing like relatively which one is better than the other. And the training engine will then up like make an edit to the model to upweight the trajectories that were more successful, and then downweight the ones that were less successful.