← All IntelClip / AI AgentsAgentic map-reduce with Harbor Exec
From Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute · ≈16:05
Demonstrates a reusable pattern: fan out a cheap agent over thousands of sessions/files, reduce with a smarter model — here mining his own coding-agent sessions for recurring mistakes to feed back into evals.
What’s in it
- Demonstrates a reusable pattern: fan out a cheap agent over thousands of sessions/files, reduce with a smarter model — here mining his own coding-agent sessions for recurring mistakes to feed back into evals.
Clip transcript
model in any sandbox on any task. And that actually doesn't mean you have to do it for evaluation. It also doesn't mean you have to do it for training. It could literally be that you want to do what we've been calling agentic map reduce which is you just want to run a ton of agents on distributed compute. of sandboxes in parallel and then somehow aggregate those results probably also intelligently. So things like looking over a bunch of trajectories to detect reward hacking or I don't know like processing a bunch of receipts for reimbursements or searching over your obsidian files to figure out uh semantically what where you wrote some note or processing a bunch of PRs to ask a question about it. So we see a bunch of different use cases. This is actually an emergent use case. So, we didn't build Harbor for this, but we see people doing it a lot. So, we actually built this feature and launched it just for this. So, it's Harbor Exec. Uh, it's going to go away kind of soon, but you can see in this scenario, actually, let me see if I can pause this. So, uh, I'm actually going to go back a little bit before we kicked it off. So, I'm saying Harbor Exec input is all of my codec sessions from the June 20s. And then I say prompt if I corrected the agent write an analysis.json with mistake reason and correction. And then my reduce prompt is summarize recurring mistakes and failure categories into a concise feedback.mmd file. And you can see in this case I'm running on modal. I do the map step with cursor CLI because it's cheap and fast. And then I do the reduce step with fable 5 because I wanted to have like an accurate summary and I care about intelligence in that scenario. So, and then I'm limiting it to 32 because I didn't actually want to process all of my sessions for this demo. But you could there's no reason you couldn't do like 10,000 sessions or a thousand sessions and then maybe you have to go and understand the results more deeply. But um Oh, shoot.
Comments
Checking sign-in…
Loading comments…