Vibeleaderboard
← All Intel
Intel / post

Epoch's Automation Reports test whether AI can do Epoch's own work

Source
x.com
Date
EpochAIResearch@EpochAIResearch

Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

Horizontal bar graph displays average task performance scores for 6 top AI models, showing that none can autonomously complete Epoch's work to internal standards.

Context

Epoch AI launched Automation Reports to test whether AI models can do Epoch's own research and communication work, and its report says they cannot yet. In the report, Epoch gave six models 11 tasks in five categories (graphic design, data insights, data explorers, AI data center research and research design), with the a new hire would get.

In its announcement, Epoch says Anthropic's Claude Fable 5.1 and OpenAI's GPT-6 Astra lead but are far from fully automating its work; the report calls them broadly tied. Epoch says leading closed models were reliable on well-defined work such as coding and computational analysis but, on open-ended tasks, often missed implicit style and audience conventions and designed weak experiments.

Epoch lists its own limits: each ran once per task, a single human grader scored each output, scores are noisy rough indicators, the tasks may not represent other jobs, and differences may reflect each 's (the software around the model) as much as the model.

Terms in this piece · Glossary
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Tool — An operation that the software around a model makes available for the model to request, with a name, expected inputs and a kind of result.
  • Harness engineering — Designing the full environment and feedback system around an agent.
More from EpochAIResearch
Recommended reads
Comments

Checking sign-in…

Loading comments…