Accelerating Reinforcement Learning Weight Refit with the Reshard Collective

Refit for the largest open models can move about 10 TB per step. Reshard transfers only the theoretical minimum data between layouts, giving 2.7x to 5.7x faster refit in NeMo RL, which cuts RL post-training overhead.

• In disaggregated RL post-training, 'refit' copies new weights to inference GPUs each step. NVIDIA says gather-and-broadcast sends about twice the model's size, roughly 11.2 TB for Kimi K3 in BF16, because each inference GPU gets full tensors when it needs only a shard.
• NVIDIA's Reshard collective treats refit as resharding a PyTorch DTensor from the training layout to the inference layout. From the two layout descriptions it computes exact shard overlaps, so only the minimum data moves even across different parallelism and GPU counts.
• The open-source NCCL M2N Reshard sends each source shard over the network once per NVLink domain, gives each replica a distinct slice, and lets replicas swap the rest over NVLink in parallel, removing the duplicate network transfers that halve per-shard bandwidth.
• In NeMo RL benchmarks on DGX H100 and DGX GB200, Reshard refit was 2.7x to 5.7x faster than gather-and-broadcast, for example 14.9 s to 2.6 s for DeepSeek v3 671B on H100. A refit-only microbenchmark moved Qwen3.8 2.4T in BF16 in 4.3 seconds.
• NeMo RL uses a hybrid: feed-forward projection weights, about 95% of parameters in recent LLMs, take the Reshard path, while small weights such as QKV or Mamba keep gather-and-broadcast for coverage. On failure recovery it currently recreates the communicator group.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

• articleScaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2
• articleAccelerating Mamba-2 with High-Performance Forward Pass Kernels in NVIDIA CUDA Tile
• articleScale Agentic Coding Workflows with Meta’s Muse Spark Open on NVIDIA GB300 NVL72
• articleOptimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym

• articleRunning Frontier-Class AI Models Across 8 NVIDIA DGX Spark NodesShara Tibken
• videoTaking Reinforcement Learning Cross Datacenter — Nan Jiang, ModalAI Engineer
• articleDeveloping Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model OptimizerTanya Lenz

Checking sign-in…

Loading comments…