Clip / Education

Causal attention, cross attention, and the softmax head

From From Transformers to Diffusion LLMs: Understanding LLaDA · ≈9:08

Walks the exact decoder path — causal attention over history, cross attention to the source, linear layer plus softmax — that every autoregressive LLM still uses today.

What’s in it

Walks the exact decoder path — causal attention over history, cross attention to the source, linear layer plus softmax — that every autoregressive LLM still uses today.

Watch this moment ↗

Clip transcript

thousands or millions. Now, we need to build an embedding for the next token to be translated. But this time, we can only leverage the tokens to the left. Everything to the right is just useless padding. This mechanism is called causal attention. It's what enables the model to look at its past. For translation, of course, we have to also consider the input sequence processed by the encoder. We'll compute a new next token embedding which is a function of the source tokens as well as the history. This mechanism is called cross attention because it crosses between languages. And we're almost there. We'll pass the final embedding through a linear layer which expands its size to one dimension per vocabulary token. The softmax layer turns it into a probability distribution from where we can sample the next token.

Intel