Argues a commonly cited barrier to scaling training clusters, hardware failure, can be mitigated by in-memory checkpointing, which affects expectations of future model scale.
Key takeaways · AI-distilled
Failure frequency scales with cluster size: at about one failure per H100 every 50,000 hours, a 16,384-GPU system fails every 3 hours on average and a million-GPU system every 3 minutes.
Epoch's models find traditional storage-based checkpointing can limit training scale if storage is not scaled up, because checkpoint overhead and lost work can surpass productive training time.
Recovering from failures using distributed memory across training peers, rather than storage, removes hardware failures as a practical limit on run size, a result the authors say holds under Chinchilla-optimal scaling.
The authors add that catastrophic failures, maintenance, and shifts toward more efficient communication patterns can change checkpointing strategy but will not limit AI scaling.
