Intel / video

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Source: AI Engineer
Author: AI Engineer
Date: Published Jul 24, 2026

Terms in this piece · Glossary

sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Why it matters

If you're building or evaluating AI agents, this reframes and optimization around 'rollouts' — connecting environments, benchmarks like Terminal-Bench, and learning loops into one practical framework from the people building Harbor and OpenThoughts-.

Key quotes

“software engineering was when you knew what the code would do before you ran it”
Alex Shaw

“agent performance itself is best treated as a blackbox artifact”
Alex Shaw

“if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models”
Satya Nadella

“you can just fire and forget and go to sleep and then come back the next day and 10,000 rollouts are done for you”
Alex Shaw

“And the answer is every single company that uses computers.”
Alex Shaw

Read the source ↗www.youtube.com