← All IntelClip / OtherFP32 to FP4: 8x Compression, Same Intelligence
From Compression at the Edge — Chris Alexiuk, NVIDIA · ≈2:39
“same cost more intelligence”
“if you quantize the correct layers you will not make the model you know literally useless.”
What’s in it
- Shows how FP4 quantization delivers 8x compression with minimal accuracy loss
- Explains dynamic quantization that shrinks a 1.5TB model down to 250GB
- Reveals why quantizing the right layers keeps 76% of model accuracy
Clip transcript
least it democratizes things. >> Cool. So the way I think about is same cost more intelligence. So compression accelerates and enables to give like u like a quick example originally we started with training in FP32 right and now we are talking about FP4 so that is 8x uh more compression and same almost same intelligence without not much degradation yeah same cost more intelligence >> yeah like how we see quantization is like you know you take like a big model like GLM 5.2 to it's 1.5 tab which is definitely ginormous. Um but then the trick is you can actually quantize it and shrink it to 250GB. Um so you can make it 86% smaller. Um but with tricks of quantization it will not become 86%. So it's not 86% dumber right if you compress it by 86% it doesn't become like you know terrible useless. Um and so what we show with dynamic quantization if you quantize some layers to you know higher precision and you leave most of the layers in like one bit or two bit and some you know very important layers is 16 bit you can still recover 76% of all accuracy. Um so you so the trick for quantization is if you quantize the correct layers you will not make the model you know literally useless. Um and compression I guess is very important for you to run on your local computers. Um you know make democratization of AI. Yeah.
Comments
Checking sign-in…
Loading comments…