Decoupled DiLoCo: A new frontier for resilient, distributed AI training
Source
deepmind.google
Date
Why it matters
Distributed training that tolerates unreliable or slow interconnects changes where large models can be trained and what hardware topology is required to do it.
articleBuilding Federated Multimodal AI Workflows with NVIDIA FLARETanya Lenz
articleTraining Variable Long Sequences with Data-Centric ParallelGeng Zhang, Xuanlei Zhao, Kai Wang, Yang You