Text Diffusion LLMs Explained: D3PM and LLaDA Paper Walkthrough
Source
Julia Turc
Author
Julia TurcTop Viber
Date
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
If you keep hearing about Gemini Diffusion or Mercury Coder and want to understand why parallel draft-refinement can be ~10x faster than autoregressive decoding, this walks through the actual formulations (D3PM's Markov chain corruption, LLaDA's masked-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → approach) instead of stopping at benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → claims. It's the level of detail needed to judge whether diffusion LLMs are worth designing around.
Key quotes
“So, ironically, diffusion leads to faster inference but slower training.”
Julia Turc
“So, in a way, BERT was a singlestep diffusion model.”
Julia Turc
“The difficulty of rounding is the reason why diffusion and token embedding space never really took off.”
Julia Turc
“There's no direct transition from dog to cat, just from dog to mask.”
Julia Turc
“But in language, because it's discreet, there's no concept of a gradient. So instead, we have to learn the landscape directly.”
