let's see how diffusion LLMs are trained. As I alluded to earlier, BERT pre-training, which uses a masked language modeling objective, is the basic building block for diffusion pre-training. However, BERT adopted a fixed masking rate of 15%. chosen empirically to maximize its capabilities as a text classifier. But a diffusion model needs to handle multiple levels of noise. Remember, my simplistic noise schedule had three thresholds, 100, 60, and 30%. Of course, if we increased the number of diffusion steps, the noise schedule would also be more granular. Going back to BERT, this is what pre-training with a fixed noise level looked like. Lada makes a simple modification. When choosing a training instance, it also picks a random noise level from the schedule. So for instance, for the parakeet article, it might choose to mask 30% of the tokens. For the cat article, it might arbitrarily pick 60%. Now, choosing a single noise level for each training instance might seem like an odd design decision at first. Why not apply all noise levels to every article just like at inference time? Well, we could do that, but implementation is a little bit easier this way. And anyway, when we train for millions of steps, chances are the same article will be selected multiple times with different levels of noise. Now, I do want to point out that diffusion training is overall somewhat inefficient. During a single training step, only some of the tokens are predicted. In contrast, GPD was able to fully leverage a training instance by predicting all the next tokens in a single pass.