Decoupled DiLoCo: A new frontier for resilient, distributed AI training

Google DeepMind's Decoupled DiLoCo divides model training among separate groups of computers that exchange updates asynchronously. This limits how much one group's failure interrupts the others. In the team's reported tests, training continued when entire groups were deliberately taken offline and incorporated them again when they returned. The team also reports training a 12-billion-parameter model across four US regions using 2–5 Gbps connections between locations. The architecture addresses both failure recovery and the communication demands of training across distant data centers.