the trade-offs it makes compared to the Fusion LLMs. Given a pre-training instance like a Wikipedia article, GPT ingests the entire input sequence at once. So, each token in the article gets its own embedding. For each position, the decoder computes an embedding for the next token. For simplicity, let's focus on one position at a time. Say the model is expected to predict dog. Then the next token embedding attends to everything up to breed of retriever. Say it wrongfully predicts the word cat. Since this doesn't match the ground truth that means the original article the model's weights will be nudged to downweight cat in the future. The same mechanism applies for the next position but this time the model prediction is correct. So the word of is reinforced in the weights. Notably all predictions can be parallelized. With a single pass through the model we're able to learn from every single token. As we'll see later, this is a lot more efficient than training the Fusion LLMs. But it does come with a drawback. The token history always comes from the training data, never from the model's past predictions. Remember, we previously predicted cat instead of dog. Yet, in the current step, we're attending to dog. Using a clean history that comes from the original article is very useful at the beginning of training when initial guesses are bad and the model would have a hard time learning anything. However, this introduces a discrepancy at inference time. All of a sudden, the model is expected to ingest its own predictions. A single wild output can derail the entire response because the model never learned how to deal with outliers. This issue called sampling drift is something that the fusion LLMs avoid altogether. So pre-training is