How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Introduces sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition →, verifier-based AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → with oracle checks before spending tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. A case study shows measuring a prompt change by tool-call count and correctness.
Terms in this piece · Glossary
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
videoWhat a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma
videoAgents That Self Improve While You Sleep - Fuad Ali, Arize AI
video300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer
videoHow a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver
videoHow to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AIAI Engineer
videoEverything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude InstituteAI Engineer
videoFrom Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, ArizeAI Engineer