hold, but also if any of these numerical issues come up again. And um of course, something happened. Um in this case, it doesn't actually have anything to do with scale, so it was just unfortunate. Um in this case, we had a race condition because we added FP8 training uh based on Deep Chem FP8 kernels that are also like open source. Um we noticed these because we hit illegal memory accesses as well as NaNs in the gradients, uh which after a while of debugging we traced back to those kernels. Um there's also an unobservable unobservable effect um that you wouldn't know about if you don't know that there's an issue. Uh in our case, we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values. Um we do have a fix available that's in a PR right now. Uh it's not been merged to Deep Chem yet, but it's public on that QR code if anyone is interested. Um and it's also an interesting point because it's a blind check and a blind spot in the hash checks. In real training runs, you don't have any redundancy where you have the same model weights and the same data, so you can never check if forward and backward actually behave the same across different model replicas. So, you can also never check if there's a race condition in that. Uh that's something that we're working on right now uh to essentially have a hash checker um that can also do that as a at least as a dry run.