From Transformers to Diffusion LLMs: Understanding LLaDA
- Source
- youtube.com
- Author
- Julia TurcTop Viber
- Date
Context
A diffusion can still be a . As Julia Turc explains in this walkthrough, the terms describe different choices: Transformer names the network architecture, while diffusion describes how the model learns to recover text and generate an answer.
The LLaDA paper makes that distinction concrete. It uses a Transformer to predict hidden text from a partially masked passage. Its can use visible material on either side, rather than restricting each position to preceding text. Training varies how much text is hidden; generation repeatedly predicts masked positions. The innovation changes the learning and generation process while retaining the Transformer machinery.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
“It starts from gibberish and iteratively turns it into a coherent response. Patterns emerge from noise.”
Julia Turc
“What the transformer did was to promote the attention mechanism from a supporting role to the main building block of the translation network.”
Julia Turc
“A single wild output can derail the entire response because the model never learned how to deal with outliers.”
Julia Turc
“there's a clear convergence towards masked diffusion models, the ones inspired by BERT”
Julia Turc
Checking sign-in…
Loading comments…