distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
Why it matters
Quantizing a 21B model to 3.5 bits costs 1.6 points with calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → error reduction and self-distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition →, compared to 6.8 with stock Q3_K_S. The library exports directly to llama.cpp Q3_K and Q4_K formats.
