inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → is now its own engineering discipline: the question is not training quality but turning weights into a fast, reliable, affordable API - and optimizations still routinely deliver 20%, 100%, even 200% gains toward 10x-faster serving.
The serving playbook in one pass: cache-aware routing asks have I seen this prefix before, prefill and decode increasingly run on DIFFERENT GPUs, speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → drafts with a small model, and KV-cache movement ties it together.
quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → errors can CANCEL each other: one GLM-5.2 experiment quantized more of the model, preserved benchmark quality, and gained 20% throughput - fidelity is a system property, not per-layer.
Models are surprisingly modular: Baseten grafted Kimi vision encoder onto GLM-5.2 without touching the language model, and retrofits inefficient layers with components from other architectures.
The loop is closing: GLM-5.2 helped optimize the very kernels that serve GLM-5.2 - training-for-inference, continual learning via persistent KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition →, and self-optimizing infra are the next phase.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Why it matters
Offers a rare practitioner-level walkthrough of how production LLM inference actually works today — cache-aware routing, disaggregated prefill/decode, quantization error cancellation, and speculative decoding — giving engineers concrete mental models for building and optimizing inference systems rather than surface-level product marketing.
Key quotes
“How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?”
“The LLM is not capable of doing anything. It's only capable of making suggestions of what to do and then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs.”
