causal attention. Now here's the interesting thing. Researchers realize that these two components are a little bit redundant. They're actually very similar to each other. So, we don't need both of them at the same time. A lot of subsequent transformer-based models use either one or the other. When using them separately, cross attention as a concept disappears. When looking at these components side by side, we can see they're actually not that different. Ultimately, they both take input embeddings and contextualize them. The main difference is the type of attention. In the encoder, everything attends to everything, while in the decoder, tokens only attend to their left. We'll now look at how subsequent models borrowed one of the two components to extend the transformer to other tasks. And we'll start with auto reggressive LLMs.