mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Why it matters
A rigorous, well-organized reference on the parallelism strategies (data, tensor, pipeline, mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition →, expert-choice routing) and memory-saving techniques that make large-scale model training feasible — useful for anyone reasoning about distributed training tradeoffs.
Key quotes
“The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory, way above what can be hosted on an individual GPU machine.”
Lilian Weng
“Bulk synchronous parallels (BSP) : Workers sync data at the end of every minibatch. It prevents model weights staleness and good learning efficiency but each machine has to halt and wait for others to send gradients.”
Lilian Weng
“The GPipe paper observed that the bubble overhead is almost negligible if the number of microbatches is more than 4x the number of partitions $m > 4d$ (when activation recomputation is applied).”
Lilian Weng
“Switch Transformer ( Fedus et al. 2021 ) scales the model size up to trillions of parameters (!!) by replacing the dense feed forward layer with a sparse switch FFN layer in which each input is only routed to one expert network.”
Lilian Weng
“The experiments showed that Gist can reduce the memory cost by 2x across 5 SOTA image classification DNNs, with an average of 1.8x with only 4% performance overhead.”
