Shows where frontier models still fail on open-ended work: designing informative experiments and trusting artifacts of their own setup. Useful calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → before delegating research-style tasks to agents.
Key takeaways · AI-distilled
Epoch built the reports to fill a gap: standard benchmarks favor easily verifiable tasks that capture only part of real work and can overstate progress for models optimized on them. Tasks come from Epoch's own work and are graded by hand against employee standards.
Models miss implicit house conventions even with ample reference material, such as Epoch's visual style and the topics its audience cares about, and tend to produce more information-dense output than Epoch's typical work.
The open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → gap shows up even on well-defined tasks: open-weight models struggle on work that Fable 5.1 and GPT-6 Astra handle reliably.
Although several tasks allowed open-ended exploration across a large idea space, the six models often converged on similar ideas.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
