Vals AI Launches Terminal-Bench 4.0 With Stricter Pass/Fail Grading

Vals AI launched Terminal-Bench 4.0, a harder version of its terminal task benchmark for coding agents. The new suite adds 66 tasks such as shipping a working service, proving a theorem, and training a GPU kernel, each estimated at roughly four hours of expert human work to complete. Grading is strict pass or fail through full verifier suites rather than partial credit, and each task runs three times with the score averaged across runs to reduce noise from lucky attempts. Earlier terminal benchmarks often rewarded partial progress or single lucky runs, which let agents post strong scores without reliably finishing real engineering work end to end. The stricter format is designed to close that gap and give a more realistic signal of whether an agent can complete multi hour tasks without human intervention. For teams evaluating coding agents, a benchmark built around longer, all or nothing tasks is a better proxy for production reliability than shorter single turn tests, though it also means scores will likely look lower across the board than on earlier, more forgiving versions of the benchmark.