So what happened? What broke this equivalence between translation and language modeling? Well, in 2018, OpenAI released their first GPT model, which stands for generative pre-training. At the time, the bottleneck for language modeling was data. There simply weren't enough response prompt pairs to be mined on the internet. In contrast, free form text like Wikipedia articles was and still is a lot more abundant. Plus, it stores a huge amount of world knowledge and linguistic patterns. To make room for it, GPTE split the training process into two stages, pre-training and fine-tuning. So, to leverage free form data during pre-training, we need to gamify it or come up with a learning objective that implicitly coerces the model to extract valuable information from the data. And that's what next token prediction is. A madeup artificial yet useful task. Given the start of an article, the model needs to predict the next word. And the same challenge repeats for every single word in the article. The next token prediction task is the reason why auto reggressive LLMs are picking the decoder side. Causal attention is a good fit by nature. When predicting the next token, it only looks at the previous ones. It's