Could Decentralized Training Solve Ais Power Problem
Source
epoch.ai
Author
Epoch AI publications
Date
Why it matters
Power limits on single data centers shape how large future training runs can get. The report shows a 10 GW run across distant sites is technically plausible but that developers will likely scale single sites to multi-gigawatt first.
Key takeaways · AI-distilled
xAI's 350 MW Colossus data center is planned to expand to 1.5 GW and OpenAI's 240 MW Abilene Stargate to 1.2 GW. At full scale, the article says, these single sites will rival the 1.2 GW Maaden and 1.6 GW Bahrain aluminium smelters.
If trends continue, training clusters are projected to reach 10 GW by the end of the decade. That would be larger than the capacity of the Grand Coulee Dam, the largest US power plant, and nearly match the total installed power capacity of all NVIDIA AI chips at the end of 2024.
NextEra CEO John Ketchum said some sites could readily support one gigawatt but it would 'take some work' to accommodate 5 GW. Mark Zuckerberg said 'we would probably build out bigger clusters than we currently can if we could get the energy to do it.'
Against the view that big pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → runs need contiguous buildouts, Epoch sketches a 10 GW run on spare capacity at 23 existing gas plants, with nearby data centers on 4,800 km of fiber using data parallelism; it cautions these plants are unlikely to lend that much.
Terms in this piece · Glossary
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
