Decoding at batch size 1 was memory-bound, so the fixes target weight reads and token acceptance. The measured stack of kernel, FP8 head and speculative decoding is reusable for local MoE inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →.
Key takeaways · AI-distilled
NVIDIA's profiling found Ling-3.0-flash decoding on DGX Spark at batch size 1 was memory-bound, with the mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → computation and the 157,184-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → output projection standing out, so each fix targets weight traffic or the number of decode steps.
Quantizing only the LM head from BF16 to FP8 halves its per-token weight reads. Combined with the Humming MoE kernels, it cut time per output token from 27.71 ms to 22.76 ms, a 21.7% throughput gain.
With the FP8 LM head and MTP speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → both on, the draft model read FP8 weights as BF16 unless it shared the full LM-head module, so accept length fell to 1.00 and throughput dropped below the unoptimized baseline.
The stacked gain of 185.1% on a synthetic workload shrank to 42.5% on all 1,319 GSM8K questions, partly because repetitive greedy outputs accepted 3.90 draft tokens per round versus 2.68 under real sampling.
On GSM8K the three changes cut mean latency by 24.5% while accuracy moved from 96.89% to 96.82%, which NVIDIA reports as no statistically detectable change.
Terms in this piece · Glossary
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
