particle from real data to noise. When diffusion happens in latent space, we're not perturbing raw pixels or tokens directly, but rather embeddings. And during inference, we do the opposite. Starting from a noisy latent, we clean it up. This process is called reverse diffusion and is equivalent to moving the particle uphill. And finally, a decoder will translate the D noiseis latent back into the original space, be it pixels or tokens. for images. This is often a VAE decoder where VAE stands for variational autoenccoder. This can be a pre-trained off-the-shelf model potentially borrowed from previous work that had nothing to do with diffusion. High level, it consists of convolutions and upsampling layers that turn the n-dimensional latent into a two-dimensional matrix of pixels. Now, here's the irony. For text, the decoder has to be auto reggressive, generating tokens left to right conditioned on the latent. So what we have here is a hybrid model that leverages both diffusion and auto reggression. This recipe is perfectly valid. Multiple papers do exactly this. But while this hybrid approach might fulfill some of the quality improvements promised by diffusion, the left to right decoder carries over some of the shortcomings as well. Which brings us to an alternative approach. The fusion in token embedding space. Instead of having a single compressed latent for the entire sequence, we could keep token embeddings separate and apply forward diffusion to each one independently. Effectively, we're preserving the token boundaries and removing the need for an auto reggressive decoder. During inference, we'll similarly apply reverse diffusion to each noised token embedding. But here's where things get tricky. The dnoised embeddings need to be mapped back to text tokens. This process called rounding turns out to be non-trivial. You see, the embeddings of our vocabulary tokens live in a vast continuous space. The D noiseis token embedding doesn't precisely correspond to any of these vocabulary tokens. The best we can do is choose the nearest neighbor, but given the sparity of the space, the neighbor could be a completely arbitrary token. The difficulty of rounding is the reason why diffusion and token embedding space never really took off. Instead, state-of-the-art models converge to a solution that might sound unintuitive at first. They diffuse text tokens directly, circumventing any numerical representations. In our landscape, the data axes are now hosting the vocabulary. Every tick corresponds to a token. Note that there's no natural ordering among the tokens. Even alphabetical sorting is arbitrary from a semantic point of view. This 2D plane encodes all possible text sequences of length two. In practice, we'd have hundreds of thousands or millions of dimensions, one for each position in the context window. In that huge hyper plane, there's one discrete point for every possible natural language text. Given the discrete nature of this space, the probability landscape is no longer a smooth mesh. It's more of a spiky terrain. We still see hill shapes with some sort of slope across the spikes. At first, the existence of such patterns might be unintuitive given that tokens are naturally unordered. But in the hyper plane, two neighboring texts differ in only one token, which means the landscape does have some notion of locality. Just like in a continuous space, data