Intel / article

Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton

Source: developer.nvidia.com
Author: Tanya Lenz
Date: Published Sep 21, 2026

Context

NVIDIA's multi-device serving integration lets an application request through one endpoint while the server coordinates several GPUs. The model must first be compiled for distributed execution; switching the server configuration does not automatically split an ordinary single-GPU model.

In NVIDIA's Cosmos 3 Nano video-generation example, mean generation time fell from 156.6 seconds on one GPU to 34.2 seconds on eight. Each configuration had five measured generations after warm-up, excluding model loading and video-file encoding. This demonstrates a way to reduce waiting by assigning more hardware to a request. The benchmark did not measure concurrent throughput or cost per video, so the speedup cannot establish a cost saving.

Key takeaways · AI-distilled

Demonstrated on Cosmos 3 Nano video generation: Ulysses parallelism splits 44,160 video tokens across up to 8 GPUs, with each CP8 rank handling 5,520 tokens outside while the 2,992-token text path stays replicated.
RPC speedup reached 6.09x at 8-way parallelism, but end-to-end latency only improved 4.58x, because roughly 10.2-10.5 seconds of fixed overhead (prompt work, scheduling, VAE decode, postprocessing) stays constant and becomes a bigger share of a much shorter total.
On one GPU the transformer accounts for 93.4% of generation time; at 8-way parallelism that share falls to 70.2%, quantifying how much of the remaining latency now sits outside the accelerated path.
Output isn't pixel-identical to the single-GPU baseline: CP8 measured MAE 16.316 / PSNR 19.400 dB against validation thresholds of MAE <=25 / PSNR >=18dB, while CP2 and CP4 stayed closer (MAE 12.759 / PSNR 21.111 dB).
The distributed graph is compiled into each TensorRT plan before deployment rather than converted at runtime; the CP8 plan's topology alone contains two initial reduce-scatters, 108 all-to-alls across 36 transformer layers, and one final all-gather.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

Read the source developer.nvidia.com