Intel / video

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Source: youtube.com
Author: AI Engineer
Date: Published Jul 24, 2026

Context

Harbor, an - framework, treats an agent's attempt at a task as a recorded experiment. Each task supplies instructions, a in which to work, a stopping condition and a verifier that checks the result. The talk walks through how these parts produce a rollout: the agent's sequence of actions together with its evaluation results.

Repeating that procedure across tasks gives developers evidence about what an agent can do when they change its model, instructions or tools. The speakers describe the verifier as programmatic tests, rubrics or an agent that inspects what happened, so the meaning of the resulting score depends on what that verifier actually checks. Harbor supplies the experiment structure; developers still have to define meaningful success.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.

Key quotes

“software engineering was when you knew what the code would do before you ran it”
Alex Shaw

“agent performance itself is best treated as a blackbox artifact”
Alex Shaw

“if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models”
Satya Nadella

“you can just fire and forget and go to sleep and then come back the next day and 10,000 rollouts are done for you”
Alex Shaw

“And the answer is every single company that uses computers.”
Alex Shaw

Read the source www.youtube.com