Vibeleaderboard
← All Intel
Intel / article

NVIDIA's Nemotron-Labs-Sigma: continuous diffusion LMs for faster decoding

Source
arxiv.org
Author
NVIDIA AI
Date

Context

NVIDIA researchers have built Nemotron-Labs-Sigma, a pair of language models with 3 billion and 8 billion parameters that write text in blocks of 32 tokens instead of one token at a time, and their paper shows the method can reach the accuracy of comparable models on math and coding tests. Its evidence on speed is narrower than NVIDIA's pitch of faster generation suggests. In the paper's timing tests, its fastest version runs at about the same speed as NVIDIA's earlier block model in tokens per second, and at 8B its answers score lower.

An ordinary language model runs once for every token it writes. A diffusion model starts a whole block as noise and refines it over several passes, so a single pass can settle many tokens at once. Most text diffusion models work like a fill-in-the-blanks puzzle, replacing masked positions with words. Sigma instead places every token as a point in a 16-dimensional space, starts the block as random points, nudges them toward the points it predicts the text should occupy, and finally reads off the most likely word at each position. NVIDIA calls this continuous diffusion, and argues that the smooth path from noise to text is easier to steer and to shorten. Sigma was trained by continuing from an existing one-token-at-a-time model's weights, and it keeps both ways of writing on one network. The main comparison is Nemotron-Labs-Diffusion (NLD), NVIDIA's fill-in-the-blanks model, matched to Sigma in architecture, training data and size.

Sigma's accuracy edge appears only at a slow setting. With 256 refinement steps per 32-token block, more model passes per token than writing one token at a time, Sigma-8B averages 71.16 across six math and coding benchmarks after , against 70.22 for NLD-8B as a diffusion model. It leads on GSM8K, 88.57 to 87.26, and HumanEval, 65.12 to 62.80, and trails on MBPP, 65.74 to 67.00. After instruction tuning, the paper's own results section calls the models on par with NLD. At 8B, Sigma's instruction-tuned model averages 59.14 across ten benchmarks against NLD's 59.92, and scores 78.4 against 84.6 on MATH-500. The abstract nonetheless says Sigma outperforms its discrete rivals on reasoning tasks such as MATH-500 after fine-tuning; at 8B the tables bear that out against older diffusion models such as LLaDA and Dream, but not against NLD.

The paper reports two ways to make Sigma faster. The first trains a student to finish a block in a few steps. With 8 passes per block, or 4 tokens per pass, the paper reports that it matches NLD and the 256-step model on GSM8K and HumanEval at both sizes, and the 3B student stays within 3% of the 256-step model on MBPP, where few steps introduce syntax errors. The second is self-: the diffusion side drafts a 32-token block in one pass, the one-token side checks it in one pass and keeps the longest prefix it agrees with, so the result is exactly what the model would write one token at a time. Run one request at a time on a single NVIDIA GB200 GPU, the 8B version tuned for drafting averages 3.46 times its own one-token speed against 3.04 times for NLD. The paper notes that each speedup is measured against that model's own one-token speed, which NLD's implementation runs about 12% faster at 8B. In answer tokens per second, that version reaches 931 against NLD's 865 on GSM8K, 620 against 652 on HumanEval and 762 against 750 on MBPP. Its returned answers score below NLD-8B's on all three: GSM8K, 86.4 to 88.8, HumanEval, 57.3 to 61.6, and MBPP, 63.8 to 66.2.

What the paper establishes is that continuous diffusion, which earlier work had tried mainly below 2 billion parameters, holds up at 8 billion on reasoning and code, and that its smooth path makes few-step distillation work without losing variety. The distilled 3B model's chance of finding a correct MBPP answer keeps rising as it tries more samples, which is the multiple-solutions use NVIDIA's announcement points to. Whether that helps reinforcement-learning training, which the authors suggest, is left to future work. All speed figures come from single requests on one GPU, and Google reported for DiffusionGemma that block generation's advantage shrinks when a server is already busy with many requests. The paper says the weights are public, so others can test these claims, but on its own figures Sigma is a research step, not yet a faster replacement for NVIDIA's existing block model.

Key takeaways · AI-distilled
  • NVIDIA's Nemotron-Labs-Sigma (3B/8B) runs Gaussian diffusion over learned 16-dimensional token embeddings instead of discrete masked tokens, warm-started from AR weights. The authors report it beats matched masked-diffusion and AR baselines, most consistently on coding.
  • The paper finds two inference knobs essential: classifier-free guidance, which drops the block's prior for the unconditional pass, and score temperature. Both passes share one forward pass at under 5% measured overhead, and combining the knobs beats either alone.
  • With distribution-matching distillation, an 8-step student (4 tokens per forward pass) matches the 256-step teacher on GSM8K and HumanEval at 3B and 8B. MBPP code picks up syntax bugs at low step counts, but the 3B student stays within 3% and keeps the teacher's output diversity.
  • Sigma keeps a causal autoregressive head on the same backbone, so it drafts a 32-token block in one call and verifies it itself, returning exactly the greedy AR output. On one GB200, the acceptance-trained 8B variant decoded up to 4.20x faster than plain AR decoding.
  • The authors saw learned embeddings collapse to about 5 effective dimensions in training, which they hypothesize causes token ambiguity at low step counts. A spherical uniformity regularizer countered the collapse and made few-step distillation markedly more effective.
Terms in this piece · Glossary
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
More from NVIDIA AI
Recommended reads
Comments

Checking sign-in…

Loading comments…