Terminal-Bench

Why it's ranked here

The rigor sits in the process around each task, not just the task list. Every proposed task gets an automated review against a 30-point rubric, adversarial runs where agents try to cheat the tests, and a human sign-off from one of two named maintainers. Leaderboard entries must cover every task with at least five trials each, count errored runs as failures, and pass an AI judge that flags reward hacking. The code is Apache 2.0, ships a citation file with a DOI, and the task authors include outside labs and individual researchers named in the repository.

What's good

The checker runs in its own container, separate from the agent, so an agent cannot quietly edit the tests that score it. Each task embeds a canary string so it can be filtered out of training data. Contributors are told to make tasks hard for real reasons, not by adding busywork or cutting network access. Leaderboard trials are copied into storage the maintainers control, so a submitted result cannot later be changed or deleted. Reports show accuracy with a 95% confidence interval plus total tokens and cost.

Tradeoffs

This is not a laptop benchmark. The guide recommends Modal or Daytona because some tasks need an H100 GPU or multiple containers, and the sample oracle check runs 500 jobs at once. The task set changes over time, so scores are only comparable within one pinned dataset version. Tasks run with open internet access, which keeps agents realistic but adds outside variability. The submission guide states community submissions are closed for the current version, with only maintainer runs added. The submission tooling requires Python 3.14 or newer.

How to use it well

Use it if you build or pick coding and operations agents and want a hard, cheat-resistant signal on long, multi-step terminal work. First run the reference solutions five times on your own sandbox to confirm every task passes before you trust agent numbers. Then run your agent and model at the default timeouts so results line up with the public board. If you have domain expertise, contributing a task is the main way in. It does not measure chat quality, short code completion, or anything that cannot be checked by a program.

Observed

License

Apache License 2.0

Runner

Harbor CLI, installed with uv tool install harbor

Recommended execution environments

Modal or Daytona cloud sandboxes; Docker for local task development

Task structure

Instruction, Docker environment, reference solution and test script per task, verifier in a separate container

Leaderboard tooling

Python CLI named lb, requires Python 3.14 or newer

Submission rule

Every task covered with at least five trials; errored trials scored as zero

Citation

CITATION.cff with Zenodo DOI