When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM
Source
developer.nvidia.com
Author
Fatima Fall
Date
Why it matters
A 464 GB model does not fit on one 252 GB GPU, so this shows how to split it across two machines behind one OpenAI-compatible endpoint. Measured throughput peaks near 608 tokens/s at concurrency 16 and falls at 32.
Key takeaways · AI-distilled
NVIDIA notes that sparse activation in a mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → model cuts compute but not memory: because any tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → can route to any expert, all of GLM-5.3 NVFP4's weights must stay resident, so the tutorial splits them across two GPUs with CPU weight offload turned off.
With tensor parallelism of 2, neither GPU holds a full copy and neither can answer alone; each token needs both ranks to exchange partial results over the RoCE link. The tutorial checks NCCL logs to confirm GPU RDMA is in use rather than a fallback to TCP sockets.
The setup turns on multi-token prediction, where the model's MTP layer drafts four tokens per step and vLLM verifies them with the full model; NVIDIA says the gain depends on how often drafts are accepted. It also stores the KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → in FP8, halving cache memory versus 16-bit.
In NVIDIA's AIPerf runs on 80 SPEED-Bench coding conversations, mean per-user throughput fell from 115.3 tok/s at concurrency 1 to 13.0 tok/s at 32, and mean time to first token rose from about 483 ms at 16 to about 899 ms at 32.
NVIDIA warns that serving on 0.0.0.0 with host networking exposes an unauthenticated API on every interface of the head Station, including management, and advises firewalling the port and adding authentication and TLS before wider access.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
