haven't chopped out 86% of its brain. >> Yeah, that's a great question. Um, so I think generally speaking, if you compress a model down by 86%, you would assume, you know, if you randomly select parts of the model to compress like delete or something like that or set set them to be like if you round it to like the closest number, you would most likely it will be not 86% dumber. It will be 100% dumber. So if you do that methodology that will not work. Um and so the main trick of language models is you should leverage the architecture of the language model itself. So language models generally have like you know 36 layers you know 50 layers many many layers each of the layers have different importance. Um and so like you know for example the most the first layer is actually very important and then the last layer is also very important but then the middle layers are kind of useless. Um and so the main reason why they're not that useful um is because when you train a language model um with like you know let's say one trillion parameters um you have to use many many tokens right so like a language model can be trained with like 30 trillion tokens um but we're still not there yet in terms of saturating all of the weights um so once you okay maybe in the future once we train to 300 trillion tokens okay maybe you can't do compression anymore okay maybe that's another topic but at the current stage the trick is um 86% of the weights do not need to be there in the model um because of the training algorithm because of you know back propagation some of the weights are very close to zero and you can literally just set them to zero um and so that's one of the tricks um and also you have to do you know layer by layer analysis you know if you quantize layer 1 what will happen to accuracy if you quantize layer two what will happen to accuracy and so on so on so on um and so you can also think of this as like a combinatorial optimization problem um you don't just do okay layer one and then do layer 2, you also have to do like you know 32 choose two layers or choose three layers. So it becomes very complicated. Um and so like you know it's a very then you get some combinatorial explosion problem you know it's not just layer by layer you know within the layer which number of this specific tensor is not quantizable or not. Um for example there is something called a um super weights there is a super weights paper which shows that if you quantize one number just one of the entire model your model becomes 20% dumber. Um, and so you need to find this specific one number and then you cannot quantize this. Um, so there's like very weird mechanisms of language models during training. Um, and yeah, there's a lot of whole research going into like quantizing models correctly. Um, yeah.