Run Terminal-Bench and other Harbor evals on Vercel Sandbox
Source
Elisabeth Rülke
Author
Elisabeth Rülke
Date
Key takeaways · AI-distilled
Passing `--env vercel` to `harbor run` executes each eval trial in its own isolated Firecracker microVM, letting evals parallelize far beyond a single local machine's capacity.
A task's network policy is enforced at the sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → firewall outside the VM, and optional credential injection attaches secrets to matching outbound requests at that firewall so they never enter the sandbox itself.
One AI_GATEWAY_API_KEY reaches hundreds of models across providers, so benchmarking a different model is just changing the `--model` flag on the same command.
The feature requires Harbor 0.22.0 or later, and Terminal-Bench is just one entry point: Harbor's registry also covers SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition →, tau3-bench, and OSWorld.
Terms in this piece · Glossary
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Why it matters
Vercel Sandbox now runs Harbor, the agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → behind Terminal-Bench, SWE-bench, and OSWorld, in isolated Firecracker microVMs, letting engineers parallelize AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → evals across models through a single AI Gateway key.
