Could Decentralized Training Solve Ais Power Problem
Source
epoch.ai
Author
Epoch AI publications
Date
Why it matters
Power limits on single data centers shape how large future training runs can get. The report shows a 10 GW run across distant sites is technically plausible but that developers will likely scale single sites to multi-gigawatt first.
Key takeaways · AI-distilled
Single sites already approach industrial extremes: xAI's 350 MW Colossus is planned to reach 1.5 GW and OpenAI's 240 MW Abilene Stargate 1.2 GW, comparable to the 1.2 GW Maaden and 1.6 GW Bahrain aluminium smelters.
A 10 GW training cluster, the scale Epoch projects by the end of the decade if trends continue, would exceed the capacity of the Grand Coulee Dam, the largest US power plant, and nearly match the installed power of all NVIDIA AI chips at end-2024.
Power supply is already a stated bottleneck: NextEra's CEO said 5 GW sites would take some work, and Mark Zuckerberg said Meta would build bigger clusters if it could get the energy.
Epoch's sketch draws on spare capacity at 23 existing gas power plants linked by about 4,800 km of fiber and coordinated with data parallelism, rejecting the conventional view that large pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → runs need one contiguous buildout.
Terms in this piece · Glossary
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
