quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
CPU inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → removes GPU supply and cost from the deployment decision for small models, and makes on-premise or edge serving practical where sending prompts to a hosted provider is not permitted.
