Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
Source
huggingface.co
Date
Terms in this piece · Glossary
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
Why it matters
Shows the recovery step after compress-then-quantize matters more than the quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition →, and that a 4-bit model can land more accurate than the 16-bit weights it came from — with the reasons QAT and QAD fall short.
