← All IntelClip / AI AgentsAnatomy of a Harbor rollout
From Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute · ≈9:20
“it's an open-source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task.”
“I think we have like three or 400 Aval sets by now. And in fact, I think two or three benchmarks even came out today that run with Harbor.”
“it is trying to be a common language for environments. So a way for people to specify things that are extremely interoperable and it allows you to maximize data velocity and just increases progress in the industry.”
What’s in it
- Explains the universal rollout loop (task → sandbox → agent → trajectory → verifier → reward) that underpins evals, SFT, and RL, and why an interoperable environment format maximizes data velocity.
Clip transcript
it to start understanding what my agent can and can't do and how to make it better. Um so that's a harbor roll out and uh so what you do is you start with your tasks and then you take it you start up your sandbox and then step one you pass that sandbox to your agent. So you're either running your agent outside the sandbox and executing commands into it or you're running it inside the sandbox and it's calling whatever its commands are as part of its program. Uh, and then it runs for some amount of time until it hits a stopping condition, produces some trajectory, and we'll come back to that later because that's important. And then you pass that sandbox to the verifier, which uh runs some verification process. And then finally, you stop the sandbox. The verifier produces a reward or a set of rewards and then you take those, you aggregate them a bunch across a bunch of rollouts in a data set with some agent, and that becomes your valid result. So this process here which looks relatively simple is actually extremely universal and uh it's also a little bit over overly simplified. So Harbor by now allows for a lot of different flavors of rollout. So you can do multi-step, you can run your verification in a separate sandbox, you can collect artifacts, you can simulate a user. Um but this is kind of like the the the bare bones approach uh that they're all built off of. So what is Harbor more specifically? One, it's a format for specifying agentic environments. Two, it's an open-source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task. And it's a registry of popular training and a val environment sets. I think we have like three or 400 Aval sets by now. And in fact, I think two or three benchmarks even came out today that run with Harbor. Um, so we're excited about that. And in general, it is trying to be a common language for environments. So a way for people to specify things that are extremely interoperable and it allows you to maximize data velocity and just increases progress in the industry.
Comments
Checking sign-in…
Loading comments…