Training Variable Long Sequences with Data-Centric Parallel

Data-Centric Parallel adjusts a training system to the length of each incoming batch instead of keeping the same configuration throughout. Long and short sequences place different demands on the machines, so a fixed setup can leave work unevenly distributed. The proposed method changes settings such as how much work runs in parallel, how many steps accumulate before an update, and whether intermediate calculations are repeated to save memory. The authors report a maximum 2.88-fold speedup in tests using 32 NVIDIA H200 GPUs. That supports adapting execution to changing sequence lengths in the tested workload; the maximum is not an expected gain for every training run.