Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Source
AI Engineer
Author
AI Engineer
Date
Why it matters
Learn Harbor's framework for evaluating agents in sandboxed environments
See how rollouts connect evals to agent optimization workflows
Meet the team behind Terminal-Bench and OpenThoughts-Agent
Transcript
Alex Shaw and Ryan Marten present a rollout-centered view of evaluating and improving AI agents. Drawing on their work on Harbor, Terminal-Bench, and OpenThoughts-Agent, they connect sandboxed environments, agent evaluations, and optimization workflows into a practical framework for generating and learning from rollouts. Speakers: Alex Shaw — Member of Technical Staff, Laude Institute Alex is the creator of Harbor, a framework for evaluating and optimizing agents and language models in sandboxed environments. Ryan Marten — Member of Technical Staff, Laude Institute Ryan builds Harbor and works on research-to-production efforts including Terminal-Bench and OpenThoughts-Agent. Harbor: https://www.harborframework.com/ GitHub: https://github.com/harbor-framework/harbor
videoWhy We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates
videoAI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j
videoPerception Agents — Antje Barth, Amazon AGI Lab
videoHarness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer
