Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Together AI researchers extended long-context training by reducing temporary working memory. Their UPipe method processes smaller groups of attention calculations in sequence and reuses the same buffers between groups. Attention relates different positions in an input; holding every group's intermediate calculations simultaneously can exhaust memory. The paper describes how this scheduling change reduces that peak.

The authors report training an eight-billion-parameter Llama model with five million tokens on a single node of eight NVIDIA H100 GPUs. That is 25% beyond the four-million-token single-node limit of FPDT, the strongest prior baseline they tested. The setup also uses recomputation and CPU memory offloading. These experiments establish training capacity and throughput, rather than how accurately a trained model understands such a long input.