Running Frontier-Class AI Models Across 8 NVIDIA DGX Spark Nodes
Source
developer.nvidia.com
Author
Shara Tibken
Date
Why it matters
Shows what eight desktop-class nodes deliver for local inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → of very large models, with measured decode scaling and topology tradeoffs. Useful when deciding between local clusters and hosted APIs.
Key takeaways · AI-distilled
Eight-node DGX Spark clusters reach 1 TB of unified memory (128 GB per node), enough for the full 753B GLM 5.3. NVIDIA measured roughly 80 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → per second of generation with GLM 5.3 Flash on eight nodes.
On dense Qwen3.8-27B (NVFP4), decode scaled from 11.86 tokens/s on one node to 52.58 on eight switch-connected nodes (4.43x), versus 40.11 (3.38x) on an eight-node physical ring. Prefill scaled only 1.94x.
RoCEnante replaces ring all-reduce with one all-to-all exchange plus local reduction, cutting communication steps for small tensor-parallel decode messages at the cost of more traffic. It needs a switch, falls back to NCCL, and NCCL stays better as scale or concurrency grows.
A switchless four-node ring is the cheapest option and matches NCCL's default ring algorithms, but splitting the cluster into groups, MPI-style all-to-all work, and the RoCEnante backend all require a switch.
With NeMo Switchyard routing easy requests to MiniCPM5 2B and hard ones to Qwen3.8 Flash Next, quality matched the Qwen-only setup (91/100 on a 92-case tool-use benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →) and aggregate throughput rose about 3x when a quarter of requests went to Qwen.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
