← All IntelClip / AI AgentsFrom rollouts to SFT, RL, and auto-optimization
From Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute · ≈18:36
Shows the same rollout artifact powers the whole improvement stack (SFT, RL via Tinker, evolutionary prompt optimization), with a growing ecosystem — Cognition migrated all evals, Poolside trains on it.
What’s in it
- Shows the same rollout artifact powers the whole improvement stack (SFT, RL via Tinker, evolutionary prompt optimization), with a growing ecosystem — Cognition migrated all evals, Poolside trains on it.
Clip transcript
my agent and pick like a better agent or train up a scale or something like that. So uh another use case of Harbor. And then we see people taking the trajectories and doing SFT. And then we also see people taking the reward or rewards the trajectory in the form of tokens and doing actual reinforcement learning. Uh for example, Tinker launched an integration with Harbor. And then uh people also do other types of optimization. So we mentioned JEA but any of these like evolutionary methods you take the text feedback in the form of a trajectory and the Aval and you can do some sort of like auto hill climbing with a harness or a scale. Um so in the last couple minutes I just want to talk about some cool things people have built with Harbor. I'll try to breeze through this. Uh so uh Frontier Suite and ultra long horizon software engineering benchmark built on Harbor Banker toolbench and investment banking benchmark built by handshake on Harbor. Um Swix saying that his team at Cognition migrated all their valves to harbor. This Rune bench this is one of my favorites. It's like a benchmark to measure how well agents can play Runescape. Um Scale launched their whole Sweet Atlas suite on Harbor. Uh, pool side uses Harbor to do all of their evaluations for model training. Uh, Kevin Goo created auto agent which is a self-optimizing agent that all you have to plug in is a Harboral. Uh, after query post train a model using Harbor Cognition released Frontier Code recently which is a Harbor benchmark and lang chain just integrated deep agents and their sandboxes into Harbor. Uh and then actually just today, Snorkel released Senior Sweet Bench, which is a benchmark for measuring how well agents can function under ambiguity with
Comments
Sign in to comment.
Loading comments…