Epoch Automation Reports: No Frontier Model Matches Employee Standard
- Source
- x.com
- Date
Can the most capable LLMs match humans at real work? Not yet, at least not on the kind of work we do at Epoch. We tested GPT-6.2 Astra and other recent models on Epoch Automation Reports, our benchmark of open-ended Epoch work tasks, ranging from making Epoch-style diagrams to designing basic research experiments. An Epoch employee scores the models’ outputs based on our internal quality standards. A score of 100% represents the standard expected of an Epoch employee. All models failed to meet the Epoch employee standard. GPT-6.2 Astra ranks fourth (averaging 65%), despite sharing the top ECI score of 174 with Fable 5.2. Fable 5.2 scored highest, averaging 82%. On one task, both Fable 5.2 and Opus 5.5 matched the level of a contractor. More on that below.

Context
Epoch AI, a research group that tracks AI progress, found that a top score on its broad capability index did not carry over to its own staff's work. On Epoch Automation Reports, a built from real Epoch tasks such as drawing house-style graphics, adding a data center to its database and designing small research experiments, no model reached 100%, the mark Epoch sets for the standard it expects of an employee. Anthropic's Claude Fable 5.2 led with an 82% average. OpenAI's GPT-6.2 Astra averaged 65% and placed fourth of six, behind Claude Opus 5.5 at 78% and Google's Gemini 4 Argon at 67%.
That 17-point gap stands out because the two models share the top score of 174 on the Epoch Capabilities Index (ECI), which combines more than 50 benchmarks into one general capability scale. Epoch reported that tie the same day. It built Automation Reports because most benchmarks test well-defined tasks with easily checked answers, while real work needs judgment. In its report, the failures are failures of judgment. Asked to test generalization with chess variants, every model used variants that already exist, which the tested models already know, so the experiment could not measure what it was meant to. Asked to write a Data Insight, one of Epoch's short articles on an AI trend, GPT-6.2 Astra chose a topic too narrow for Epoch's readers, and its chess variants report led with a finding it later contradicted. Models mostly kept their first attempt instead of revising it.
One task crossed a practical line. In AI data center analysis, a model researches news and regulatory documents, orders satellite images and estimates a site's chips and computing power. Fable 5.2 scored 98% and Opus 5.5 88%, and Epoch's data center analyst judged both comparable to the contractors Epoch uses, whose work also usually needs some revision. A contractor spends about 16 hours per data center. Opus 5.5 took just under 3 hours at lower cost, while Fable 5.2 took about 6.5 hours and cost more than a contractor. Epoch says it is now considering shifting contractor time toward reviewing AI output. GPT-6.2 Astra scored 65% on that task, partly because it covered only one of several AI buildings its sources pointed to and left out estimates it had enough information to make.
Epoch treats the percentages as rough. Each model ran once per task, Epoch staff graded the outputs with subjective rubrics, the tasks reflect Epoch's own work, and each model ran in its maker's tool, such as Codex or Claude Code, which may explain some differences. Epoch says its qualitative findings are more informative than the exact numbers, and those point the same way: data center analysis is nearing automation, while design and research tasks lag because of weaknesses in judgment and presentation.
Checking sign-in…
Loading comments…




