It tests AI on real recurring job tasks, such as graphic design and research planning, against the standards applied to human work. This shows where benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → scores overstate real-world capability.
Key takeaways · AI-distilled
Epoch's Automation Reports grade AI on tasks its own staff do, such as graphic design and planning research projects, using the same rubric applied to human work. A score of 100% would mean the task can be handed to AI.
The leaders, Claude Fable 5.1 and GPT-6 Astra, met 65% of rubric items on average. Epoch counts that as a failed task, since a human submission at that level would be rejected or revised, and says that does not necessarily mean AI is 65% of the way to automation.
Models did better on clearly defined work like coding and data analysis than on open-ended tasks that involve judgment. In one pilot, GPT-6 Astra set the evaluated models' token budgetA cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.Full definition → so low that 22% of runs returned nothing, then reported those failures as a capability finding.
Epoch removed a basic chart-generation task because Fable 5.1 produced an acceptable chart in one try most of the time, and now partly automates that step. Epoch expects more economic impact as models cross the human-quality threshold on more tasks.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
