Intel / video

What's New in Inference Engineering — Philip Kiely, Baseten

Source: youtube.com
Author: AI Engineer
Date: Published Sep 19, 2026

Context

Training can make an existing language model faster to run by improving the smaller model that drafts its next output. In , a draft model proposes several text units and the main model checks them together. More accepted proposals mean more output per checking step, although drafting also consumes computing resources. Baseten explains this tradeoff in its inference engineering guide.

The company's talk describes inference optimization increasingly becoming a training problem. Its live-training account supplies a concrete example: internal representations from requests being served feed a separate training pipeline that updates the drafter. This lets the drafter adapt to actual traffic and changes in the main model, making the match between the two models an ongoing engineering task.

Key takeaways · AI-distilled

The talk's framing came from a real incident: TurboQuant reached 20 million people and dipped a memory-stock index because people assumed the had halved, prompting Kiely's team to run the actual bandwidth-vs-compute math.
KV cache (distinct from ) uses a learned bottleneck that cross-attends fixed query vectors against the full KV cache to emit compact keys and values in a single forward pass.
Same-family small draft models made poor speculative decoders; training a drafter directly on the target model's hidden states worked far better, and a newer diffusion drafter proposes 8-16 tokens per step.
Days before the talk, a method paired the diffusion drafter with a sequential one, and continuously retraining the speculator on live prompts lifts acceptance 20% to 2x, if a team can afford the storage, compute, and permissions.
Kiely frames local inference work as 'get it working, then make it less dumb' versus data-center work as 'get it working, then make it less slow' -- and says data-center optimizations increasingly come out of a dedicated training process, blurring training and inference.

Terms in this piece · Glossary

speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Read the source www.youtube.com