• The talk's framing came from a real incident: TurboQuant reached 20 million people and dipped a memory-stock index because people assumed the KV cache had halved, prompting Kiely's team to run the actual bandwidth-vs-compute math.
• KV cache compaction (distinct from quantization) uses a learned bottleneck that cross-attends fixed query vectors against the full KV cache to emit compact keys and values in a single forward pass.
• Same-family small draft models made poor speculative decoders; training a drafter directly on the target model's hidden states worked far better, and a newer diffusion drafter proposes 8-16 tokens per step.
• Days before the talk, a method paired the diffusion drafter with a sequential one, and continuously retraining the speculator on live prompts lifts acceptance 20% to 2x, if a team can afford the storage, compute, and permissions.
• Kiely frames local inference work as 'get it working, then make it less dumb' versus data-center work as 'get it working, then make it less slow' -- and says data-center optimizations increasingly come out of a dedicated training process, blurring training and inference.

• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
• quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
• context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.

Baseten's Philip Kiely explains why 4-bit KV cache quantization only makes sense on memory-starved local machines rather than in data centers, and describes a diffusion-based speculative decoder that more than tripled token acceptance in production.

• videoLarge clusters for small models — Daniel Svonava, Superlinked
• videoWeight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli
• videoThe Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
• videoTwo Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

• articleThe Inference Engineering MasterclassLatent Space
• videoKV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red HatAI Engineer
• videoDeep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI ResearcherAI Engineer

Checking sign-in…

Loading comments…