← All IntelClip / EducationDiffusion LLM hyperparameters: steps, noise schedule, remasking
From From Transformers to Diffusion LLMs: Understanding LLaDA · ≈19:42
Gives the practical control surface of a diffusion LLM — diffusion step count trades speed for quality, and low-confidence tokens can be selectively remasked.
What’s in it
- Gives the practical control surface of a diffusion LLM — diffusion step count trades speed for quality, and low-confidence tokens can be selectively remasked.
Clip transcript
constantly changing and Lada will probably not be the one to establish the status quo. When designing the Fusion LLMs, there are a few important hyperparameters. Just like in traditional LLMs, there's a context window. I'm arbitrarily setting it to 35 here, but in practice, it's hundreds of thousands or millions. At the start of inference, we'll populate it with the prompt followed by the special beginning of sequence token. For masked diffusion models in particular, the rest is filled with masks. In other words, the current noise level is 100% since the entire response is full of noise. A second hyperparameter is the number of diffusion steps. That is how many times we'll refine the response draft. I'm setting it to three because I'm lazy, but in practice, it's somewhere in the tens or low hundreds. This acts as a knob between speed and quality. And finally, we need a noise schedule, which dictates how many masks we'll uncover at each diffusion step. It's common for language models to use a simple linearly decreasing schedule, like I did here. So, at step one, all masks are uncovered in one go. Because it's so early in the process, most guesses are bad, perhaps with the exception of the token 42. So, we'll throw the dice again, this time for only 60% of the tokens. The tokens to be remasked can be chosen at random or using some huristic like selecting those tokens that were assigned low confidence by the LLM. The next reveal improves the response quite a bit. Three tokens are now on the right track. It's also interesting that this time the special end of sequence token was generated. This signals that the intended model response goes all the way to hollow planet and everything after it should be ignored. We'll continue with 30% noise mask and unmask. This was our last step. So the final answer is 42 give or take.
Comments
Checking sign-in…
Loading comments…