How to Train Really Large Models on Many GPUs?

Adding more graphics processors does not by itself solve the problem of a model too large for one processor's memory. Lilian Weng's distributed-training survey explains why: ordinary data parallelism copies the entire model onto each worker and divides the training examples between them. Every worker still needs room for its copy.

Model parallelism instead divides the model across workers. Splitting consecutive layers creates a new scheduling problem, because later stages must wait for earlier ones. Pipeline parallelism reduces that idle time by dividing a batch into smaller groups that different stages can process simultaneously. The arrangement must address both where the model fits and how to keep the processors occupied.

“The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory, way above what can be hosted on an individual GPU machine.”

Lilian Weng

“Bulk synchronous parallels (BSP) : Workers sync data at the end of every minibatch. It prevents model weights staleness and good learning efficiency but each machine has to halt and wait for others to send gradients.”

Lilian Weng

“The GPipe paper observed that the bubble overhead is almost negligible if the number of microbatches is more than 4x the number of partitions $m > 4d$ (when activation recomputation is applied).”

Lilian Weng

“Switch Transformer ( Fedus et al. 2021 ) scales the model size up to trillions of parameters (!!) by replacing the dense feed forward layer with a sparse switch FFN layer in which each input is only routed to one expert network.”

Lilian Weng

“The experiments showed that Gist can reduce the memory cost by 2x across 5 SOTA image classification DNNs, with an average of 1.8x with only 4% performance overhead.”

Lilian Weng