Vibeleaderboard
← All Intel
Intel / article

Reducing Rollout Bottlenecks in Reinforcement Learning with NVIDIA Dynamo

Source
developer.nvidia.com
Author
Amr Elmeleegy
Date

Context

NVIDIA is pitching Dynamo, its software for spreading model requests across many GPUs, as a fix for a slow stage of reinforcement learning (RL) training (NVIDIA's post). In RL, a model improves by being rewarded for successful results, and the training loop depends on . Workers run the current model to generate rollouts, sequences of outputs, actions and environment interactions such as a coding 's edits, tool calls and tests. The RL framework scores them, updates the model's weights and sends the new weights back to the workers for the next round.

NVIDIA names three constraints on that loop, and Dynamo sits between the RL framework and the inference engines to address them. Rollouts vary in length, and when training needs a group of samples the slowest one holds up the next step, so spreading requests evenly across GPUs does not spread the work evenly. Many attempts also start from the same instructions or repository . Dynamo's router sends such requests to a worker that has already processed and cached that shared prompt, while avoiding piling so many onto one worker that a queue forms. Finally, workers need fresh weights throughout training. ModelExpress, a companion component NVIDIA described in an August post, copies weights already on another GPU in the cluster instead of reloading them from remote storage, and between updates sends a compressed record of what changed instead of the full model.

Dynamo also detects failed workers, sends new requests to healthy ones and adds replacements once they are ready, and a replacement can load a checkpoint and apply the later weight changes to catch up. The split of duties matters to teams with an existing RL setup: the RL framework keeps control of rewards, which samples count toward training, and when workers apply new weights. NVIDIA lists integrations with the open-source RL frameworks verl, NeMo RL, Slime and prime-rl.

The performance evidence in the post comes from adopters, not from a NVIDIA ran. NVIDIA says H Company used Dynamo and ModelExpress to cut average training-step time by up to 20% and GPU-hours by up to 5%, and that CoreWeave uses them to speed up weight updates by up to 15 times. It also lists Microsoft AI, Cognition and Prime Intellect as users. These are best-case figures without stated baselines, so they show where gains can come from rather than what a given training run will see.

Key takeaways · AI-distilled
  • NVIDIA positions Dynamo as the serving layer between reinforcement-learning rollout orchestration and inference workers. The RL framework keeps rewards, environments, checkpoints and sample acceptance; Dynamo handles routing, worker management, serving controls and telemetry.
  • NVIDIA notes that even request counts do not mean even work: a reasoning rollout may be one response, while a coding-agent rollout can involve repository exploration, edits, tool calls and tests. The slowest required rollout can delay the next training step.
  • Dynamo's KV-aware router balances prompt-cache reuse against worker load. Sending every attempt that shares a prompt to the worker holding its cache avoids recomputation, but sending all of them there builds a queue that delays rollouts.
  • Its ModelExpress component reduces weight-update overhead by copying weights already in the cluster GPU to GPU over NIXL and by sending compressed deltas against a matching base version. A replacement worker can load a checkpoint and apply later deltas to catch up.
  • NVIDIA cites adopters: CoreWeave uses Dynamo and ModelExpress to speed weight updates by up to 15 times, and H Company reports up to 20% lower average training-step time and up to 5% fewer GPU-hours. These figures are vendor-reported.
Terms in this piece · Glossary
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Read the source developer.nvidia.com
More from Amr Elmeleegy
Recommended reads
Comments

Checking sign-in…

Loading comments…