Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
Model weights are billions of numbers, normally stored at 16-bit precision. Quantization stores them in 8, 4, or even fewer bits — a quarter of the memory or less — which is often the difference between needing a datacenter GPU and running on a laptop.
The surprise is how little quality it costs when done well: 8-bit is usually indistinguishable, 4-bit is the sweet spot for local models. Degradation shows up first in the hardest tasks — long-chain reasoning and precise code — which is why quantized-model benchmark claims deserve scrutiny.