Hardware Failures Wont Limit Ai Scaling

This 2024 analysis from Epoch AI, by Alexander Erben and Ege Erdil, asks whether GPU failures put a ceiling on how big a training cluster can get, and concludes that they should not, provided labs move checkpoints out of storage and into GPU memory (epoch.ai). The worry is simple arithmetic. Using a failure rate of about once per 50,000 hours per H100, derived from Meta's Llama 3.1 405B training run, a 100,000-GPU cluster would see a failure every 30 minutes and a million-GPU cluster one every three minutes. Training survives failures by checkpointing, periodically saving its state so it can restart from the last save. Checkpoint too often and the run wastes time saving; too rarely and each failure throws away more work.

The authors model that tradeoff and apply it to Llama 3.1 405B, estimating that about 2.1% of training time went to checkpointing and recovery, with optimal checkpoints every four minutes. The constraint in Meta's setup was storage bandwidth: writing the optimizer state took about 2.5 seconds. Holding the model and everything else fixed, they estimate failures would overwhelm training only at around 70 million GPUs. The proposed alternative is to copy each GPU's checkpoint into the spare memory of other GPUs in the cluster and recover from those copies, a strategy the authors note Google DeepMind described using while training Gemini. Keeping four copies makes losing every copy at once extremely unlikely, and checkpointing over the GPU network rather than to storage pushes the limit for the same model to roughly 560 million GPUs.

Because a lab with a far larger cluster would train a larger model, the authors also redo the calculation with model size growing along Chinchilla scaling laws. The limit then lands between tens of billions and sextillions of GPUs, depending on how network bandwidth scales with cluster size, which they treat as no practical bound. The conclusion rests on failure rates staying near once per 50,000 GPU-hours and on roughly Chinchilla-style scaling. The authors also say significant engineering is still needed, since memory-based checkpointing had at that point been demonstrated only by Google, and that disasters like power outages or fires still call for slower checkpoints to remote storage.