Intel / article

Run Terminal-Bench and other Harbor evals on Vercel Sandbox

Source: vercel.com
Author: Elisabeth Rülke
Date: Published Sep 17, 2026

Context

Vercel can now run evaluations built on Harbor, the open-source behind Terminal-Bench, whose registry also includes benchmarks like , tau3-bench, and OSWorld (vercel.com). Passing --env vercel to harbor run executes each trial in its own isolated Firecracker microVM, letting a team parallelize far beyond what a single local machine could handle, while a task's network policy is enforced at the sandbox firewall, outside the VM, where optional credential injection attaches secrets to matching outbound requests without exposing those secrets inside the sandbox. Paired with Vercel's AI Gateway, one API key reaches models from multiple providers, so benchmarking a different model is the same command with a different --model flag.

Key takeaways · AI-distilled

Passing `--env vercel` to `harbor run` executes each eval trial in its own isolated Firecracker microVM, letting evals parallelize far beyond a single local machine's capacity.
A task's network policy is enforced at the sandbox firewall outside the VM, and optional credential injection attaches secrets to matching outbound requests at that firewall so they never enter the sandbox itself.
One AI_GATEWAY_API_KEY reaches hundreds of models across providers, so benchmarking a different model is just changing the `--model` flag on the same command.
The feature requires Harbor 0.22.0 or later, and Terminal-Bench is just one entry point: Harbor's registry also covers SWE-bench, tau3-bench, and OSWorld.

Terms in this piece · Glossary

sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Read the source vercel.com