um, actual pre-training side rather than just data. I liked in the previous talk, um, the the speaker made a point that we should treat different data mixes holistically, different training stages. I want to make the same point that we should treat uh, data and implementation of your training code base, uh, correctness of it, and so on, also holistically. If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't. Um, so I specifically focus on architecture work on and distributed training and so on. Um and the way we we look at things in my team is uh we don't trust anything. There's so many things that can go wrong when you scale models to billions of parameters to hundreds of billions of parameters um training on thousands of GPUs and so on. And I want to show you some of the learnings that we got from training Laguna m.1. Um And yeah, some of the surprising things that happen at scale. Um so one thing we do is we've got these um model replica hashtags. So essentially when we train a model we've got multiple replicas of the same model, right? Um distributed data parallel. And uh we know there's an invariant, the weight should always be the same across all of these replicas. That's something you can verify, right? We you can calculate a hash over the weights and you know that should always be the same across all replicas. So we do that in training and periodically compare them. Um if all of these hashes are identical, then we know we can continue training. If they're not identical, we know something has gone seriously wrong uh because that should never happen. And we crash the training. And I'll give you some examples now of things that we've not shared before publicly like this. Um so I hope they're interesting. Um so the first example here of uh that happens at scale are broken GPUs. On the left-hand side we've got two loss curves uh and on the right-hand side the corresponding uh gradient norms um that we observed during training. And you can see that these loss curves look quite different, right? Like the purple one has got quite some bumps, looks a bit spiky. The gradient norms are huge for that run. And there's actually no difference uh in terms of model configuration, training data, training implementation um between these runs. They're exactly the same run. Just in one of them we were got unlucky and we had a broken GPU included. That broken GPU caused silent data corruption and um therefore made the training behave the way it did. Uh and that is one of those cases that you can catch with these hashtags because you know this computation should be the same across all replicas, but it wasn't. Um Which brings me to the next uh instance of that happens at scale. Uh in this case exploding gradients. Um again, we're looking at two different loss curves and the corresponding gradient norm curves. Um, the purple run is our initial training run for Laguna M 1. We're a bit further into training here around 50,000 steps or so. And you can see it stops converging, right? Like it just flattens out. And the reason here was that during training the uh activations grew and grew um right before the um LM head, the unembedding. And um we have to perform some sort of accumulation here uh because we use tensor parallel for the uh unembedding. And that accumulation um was performed in BF16 by default. And because of the growing scale that we observed in the activations, um there wasn't enough um numerical precision available anymore um to do this accurately. And hence the model just couldn't learn anymore. And this is also very uh dramatic point for this to happen because it from there on it really like back propagates into the full model trunk. Um, the orange curve is essentially just adding a fix on that. So, we took the checkpoint from the purple curve. We moved that accumulation into FP32 and from there on the model started converging again. Um, the gradient norm, as you can see, actually started decreasing. Um, before then we had an increasing trend. And um this is also something you can only observe at scale. And that will break your model if you if you're not careful about it. So, as Mara said, we took all of these