Intel / post

Vals AI Launches Terminal-Bench 4.0 With Strict Pass/Fail Grading

Source: x.com
Date: Published Sep 16, 2026

From the Daily Brief

Vals AI launched Terminal-Bench 4.0, a harder version of its terminal task benchmark for coding agents. The new suite adds 66 tasks such as shipping a working service, proving a theorem, and training a GPU kernel, each estimated at roughly four hours of expert human work to complete. Grading is strict pass or fail through full verifier suites rather than partial credit, and each task runs three times with the score averaged across runs to reduce noise from lucky attempts. Earlier terminal benchmarks often rewarded partial progress or single lucky runs, which let agents post strong scores without reliably finishing real engineering work end to end. The stricter format is designed to close that gap and give a more realistic signal of whether an agent can complete multi hour tasks without human intervention. For teams evaluating coding agents, a benchmark built around longer, all or nothing tasks is a better proxy for production reliability than shorter single turn tests, though it also means scores will likely look lower across the board than on earlier, more forgiving versions of the benchmark.

Read the 2026-09-17 Brief →

Context

A Terminal-Bench 4.0 score on Vals AI's leaderboard counts only finished jobs. Vals AI, which runs AI models through benchmarks and publishes the results on its site, added the benchmark on September 16. Its 66 tasks each ask an working in a command-line terminal to carry a project all the way through, such as shipping a working service, proving a theorem, training a GPU kernel or writing a forensic report, and Vals estimates the median task at four hours of expert work. A task counts only if the agent's result passes every check in that task's verifier suite, so getting most of the way earns nothing. Each model attempts every task three times and the published number averages those three single-attempt scores, so it reflects how often a model finishes on a typical try rather than its best run.

That all-or-nothing rule, applied to tasks this long, helps explain why the scores look low. On the chart Vals posted with the announcement, OpenAI's GPT-6 Astra leads at a little under 60 percent, Anthropic's two best entries sit roughly in the mid to high 40s, and every other model plotted, including entries from Google, DeepSeek, Alibaba and Meta, completes less than about 30 percent of tasks. Vals was not the first to publish results: Anthropic had already charted Terminal-Bench 4.0 results in its Claude Fable 5.1 release earlier in September. Publishers can run models with different agent setups, so a Vals number compares most cleanly with other Vals numbers.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Read the source x.com