Text Diffusion LLMs Explained: D3PM and LLaDA Paper Walkthrough
- Source
- youtube.com
- Author
- Julia TurcTop Viber
- Date
Context
Text diffusion can learn to restore missing text without treating words like noisy pixels. In Julia Turc's explanation, one approach corrupts a training passage by replacing text units, called , with mask markers. The model learns to recover the hidden content from what remains.
Generation reverses that setup: an answer begins as masks, and repeated prediction steps progressively fill it in. Several positions can be predicted together, with visible text providing for later steps. This explains how diffusion applies to discrete language: the corruption can be missing symbols rather than numerical noise. The number of refinement steps then becomes a choice affecting generation speed and quality.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
“So, ironically, diffusion leads to faster inference but slower training.”
Julia Turc
“So, in a way, BERT was a singlestep diffusion model.”
Julia Turc
“The difficulty of rounding is the reason why diffusion and token embedding space never really took off.”
Julia Turc
“There's no direct transition from dog to cat, just from dog to mask.”
Julia Turc
“But in language, because it's discreet, there's no concept of a gradient. So instead, we have to learn the landscape directly.”
Julia Turc
- 00:00Intro
- 01:04Auto-regressive vs diffusion LLMs
- 02:06Why bother with diffusion for text?
- 06:30The probability landscape
- 07:57Diffusion in latent embedding space
- 11:00Diffusion in token embedding space
- 12:13Diffusion in text token space
- 13:49Markov chains
- 16:46Paper study: D3PM
- 19:42Paper study: LLaDA
- 22:30Evaluation
Checking sign-in…
Loading comments…