Vibeleaderboard
← All Intel
Intel / video

What Makes Open Models Fast in Production — Sujee Maniyam, Nebius

Source
youtube.com
Author
AI Engineer
Date

Context

A smaller model can help a larger model generate answers faster by proposing text for it to check. This technique, , is one of the optimizations discussed in Nebius's production-inference talk, which also describes training custom draft models on your own traffic.

NVIDIA's technical explanation, also covered in earlier Intel, explains the mechanism: the small model proposes several tokens, and the larger model verifies them in parallel. Useful speedup depends on how many drafted tokens are accepted and the cost of drafting and verification. This explains why serving performance depends on the surrounding system as well as the main model.

Key takeaways · AI-distilled
  • Nebius's Dylan Bristot and Sujee Maniyam argue open models have nearly caught up with proprietary ones, so the hard part is serving them fast and cheaply, framed against closed APIs on cost and lock-in.
  • Their serving stack starts at hardware and engine choice (NVFP4 on recent NVIDIA GPUs) and adds cache-aware routing, which they contrast with naive load balancing for LLMs.
  • One optimization they cover is speculative decoding with custom draft models trained on your own traffic.
  • Further levers covered: offloading, disaggregating prefill from decode, and finding the sweet spot.
Terms in this piece · Glossary
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
  • quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…