lowerbound. To get a concrete understanding of how training and inference work in a text diffusion model, we'll look at Lada which was published earlier this year. Here are the three stages in the model lifetime. Pre-training, instruction fine-tuning, and sampling. During pre-training, we start with a general purpose piece of text, mask part of it, and then predict the masks. Interestingly, we randomly choose a single level of noise between zero and one. In this paper, they use the variable t for time, but what they actually mean is beta t for noise. They're basically conflating time with noise, which is a confusing but common practice in the fusion papers. Drizzing a single level of noise makes training inefficient. Here they're only predicting two out of eight tokens. So the remaining six are an opportunity cost. In contrast, an auto reggressive model would have predicted all eight tokens in a single training forward pass. This other paper shows that diffusion models require 16 times the amount of training compute to achieve the same validation loss. So, ironically, diffusion leads to faster inference but slower training. This pre-training strategy might remind you of BERT, one of the very first LLM encoders. Similarly, BERT masked and predicted part of its input tokens. The difference was it only used a single fixed noise level, namely 15%. So, in a way, BERT was a singlestep diffusion model. Moving on to instruction