Clip / Education

Text diffusion is converging on masked diffusion models

From From Transformers to Diffusion LLMs: Understanding LLaDA · ≈18:56

A concrete read on where an unsettled research area is heading, plus the reason: masked formulations are easier to understand and implement than math-heavy alternatives.

What’s in it

A concrete read on where an unsettled research area is heading, plus the reason: masked formulations are easier to understand and implement than math-heavy alternatives.

Watch this moment

Clip transcript

diffusion LLMs. Text diffusion is still a very young field. As I covered in my previous video, there are many formulations, some more math heavy than others. And while there isn't a de facto standard yet, two things are clear. One is that all diffusion MLMs under one form or another are using the transformer. and two is that there's a clear convergence towards masked diffusion models, the ones inspired by BERT. Most likely, it's because they're easier to understand and easier to implement. To make it