🧬 Introducing LLaDA2.0, for the first time scaled to 100B, as a Discrete Diffusion LLMs (dLLM)!
Featuring 16B (mini) and 100B (flash) MoE versions.
With 2.1x faster inference than AR models and superior performance in Code, Math, and Agentic tasks, we prove that at scale, Diffusion is not just feasible—it's stronger and faster. 🌊
#AI #LLaDA #Diffusion #OpenSource #dllm
Why LLaDA2.0 can be a game-changer?
🚀 100B Scale: The largest dLLM to date. Declared its potential to scale (finally!).
⚡ Blazing Speed: Achieves 535 tokens/s (2.1x vs AR) via parallel decoding, merged in SGLang.
🔄 Cost-Effective: Specific training strategies allow seamless knowledge transfer from existing AR models.
🔍 Fully Open Source: Full opened weights & code for both 16B/100B available now on HuggingFace.
Core Tech - WSD Strategy
How did we bridge the gap between AR and Diffusion? Introducing Warmup-Stable-Decay (WSD):
🔹 Warmup: Gradually increase block size, transitioning from AR mode to full-sequence context.
🔹 Stable: Train on massive corpora with Masked Diffusion to master bidirectional understanding.
🔹 Decay: Shrink block size (e.g., to 32) to optimize KV Cache usage and inference efficiency.
Post-Training Alignment Stack
Conversion alone doesn't create a usable assistant. Let's see what we did through post-training for LLaDA2.0:
👨🏫 Supervised Fine-Tuning: Complementary masking ensures 100% token utilization; mask ratio bandwidth stabilizes convergence
🎖️ DPO over ELBO: Direct Preference Optimization reformulated using Evidence Lower Bounds, trained on 1.5M preference pairs
🎯 Confidence-Aware Training: Auxiliary loss sharpens predictions for efficient parallel decoding (threshold 0.95)
🚀 Fine-tuning framework, dFactory: https://t.co/qBXcOZI6wX
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
A discrete diffusion LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → at 100B with open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → and parallel decoding at 535 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →/s makes diffusion a practical alternative to autoregressive serving, and the WSD recipe shows how existing AR models can be converted rather than trained from scratch.
Read the full thread on X ↗
