What Makes Open Models Fast in Production — Sujee Maniyam, Nebius
Source
youtube.com
Author
AI Engineer
Date
Context
A smaller model can help a larger model generate answers faster by proposing text for it to check. This technique, speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition →, is one of the optimizations discussed in Nebius's production-inference talk, which also describes training custom draft models on your own traffic.
NVIDIA's technical explanation, also covered in earlier Intel, explains the mechanism: the small model proposes several tokens, and the larger model verifies them in parallel. Useful speedup depends on how many drafted tokens are accepted and the cost of drafting and verification. This explains why serving performance depends on the surrounding system as well as the main model.
Key takeaways · AI-distilled
Nebius's Dylan Bristot and Sujee Maniyam argue open models have nearly caught up with proprietary ones, so the hard part is serving them fast and cheaply, framed against closed APIs on cost and lock-in.
Their serving stack starts at hardware and engine choice (NVFP4 on recent NVIDIA GPUs) and adds cache-aware routing, which they contrast with naive load balancing for LLMs.
One optimization they cover is speculative decoding with custom draft models trained on your own traffic.
Further levers covered: KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → offloading, disaggregating prefill from decode, and finding the quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → sweet spot.
Terms in this piece · Glossary
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.