The paper estimates that data movement limits training scale beyond about 2e28 FLOP, with a latency wall near 2e31 FLOP, reachable in roughly three years. It names aggressive batch size scaling as a possible escape, which bears on how far frontier training can grow.
Key takeaways · AI-distilled
Epoch estimates frontier training compute grew 4 to 5x a year from 2010 to 2024, far faster than Moore's law, by making runs longer, using more GPUs per run, and using faster GPUs.
Runs rarely exceed about 6 months, since hardware and software risk going obsolete and no lab wants a model outdated at release, so growth has come mostly from bigger clusters: the original transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → used 8 GPUs for 12 hours, Llama 3.1 405B about 16,000 for 2 months.
The 2e28 FLOP limit assumes a run of at most 3 months: beyond that scale, the time spent moving data within and between GPUs dominates the time spent on arithmetic, so hardware can no longer be used efficiently.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
