Training Variable Long Sequences with Data-Centric Parallel
Source
Geng Zhang, Xuanlei Zhao, Kai Wang, Yang You
Author
Geng Zhang, Xuanlei Zhao, Kai Wang, Yang You
Published
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
Teams fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → or training on variable-length data get a low-friction way to recover throughput lost to static parallel configurations and workload imbalance.
Transcript
Training deep learning models on variable long sequences poses significant computational challenges. Existing methods force a difficult trade-off between efficiency and ease-of-use. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models. To break this trade-off, we introduce Data-Centric Parallel (DCP). Its core principle is to let the data itself drive the runtime. It achieves this by dynamically adjusting direct runtime settings (e.g., parallel size, gradient accumulation, recomputation) based on each batch's sequence length. Empirical results demonstrate that our method achieves up to a 2.88$\times$ speedup on 32 H200 GPUs. Designed for generalization, it can be integrated into any model with 10 lines of code. We anticipate this simple yet effective approach will serve as a robust baseline and facilitate future advancements in distributed training for variable long sequences.