Long-horizon agents are usually graded pass/fail. This benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →'s progress ladders and held-out scenarios measure sustained progress, so partial gains show up in comparisons.
Key takeaways · AI-distilled
On the 30 reconstruction tasks, binary verifier scores averaged 0.221 while dense progress-ladder scores averaged 0.653, which the authors read as evidence that pass/fail grading discards most intermediate progress.
Reported leaders across 1,145 scored runs from 15 models: GPT-6-astra with dense mean reward 0.69 and 37 tasks at R of at least 0.95, then GPT-5.6-sol at 0.67 and 32 tasks.
Version 2.0 extends the horizon from 90 minutes to three hours with verifier checkpoints every 30 minutes. New tasks reveal six generated scenarios as evidence and hold out three for scoring to prevent memorizing public outputs.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
postTerminal-Bench-Science leaderboard debuts: GPT-6 Astra and Claude Opus 5.5 leadArtificial Analysis
articleE2E-SWE: Benchmarking LLMs on Building Working Codebases from ScratchHantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
videoStanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon TasksStanford Online