Clip / Education

BERT's masked language modeling objective

From From Transformers to Diffusion LLMs: Understanding LLaDA · ≈16:53

Masked language modeling is the direct ancestor of masked diffusion LLMs, and the fixed 15% rate is the specific thing diffusion models had to generalize.

What’s in it

Masked language modeling is the direct ancestor of masked diffusion LLMs, and the fixed 15% rate is the specific thing diffusion models had to generalize.

Watch this moment

Clip transcript

therefore the encoder is a natural fit. All tokens can attend to all the others. But pre-training gets a bit trickier with an encoder. The next word prediction task is no longer supported since tokens peak into the future. The solution came from BERT, a pre-trained text classifier published by Google in 2018. BERT came up with a new task for pre-training, masked language modeling. Exactly 15% of the tokens are replaced with a special mask token and the model needs to recover them all in one go. Let's see how to pre-train the transformer encoder with the help of the masked token prediction test. As usual, the encoder produces a contextual embedding for every input token, including the masks. To turn these mask embeddings into token predictions, we can borrow the same trick we used in the decoder, a linear layer followed by softmax. Note that mask predictions can be paralyzed since they're independent from each other. The model predictions are