What Makes Open Models Fast in Production — Sujee Maniyam, Nebius

Lists the specific serving layers that determine cost and latency for self-hosted open models, which helps when weighing a closed API against running open weights.

• speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
• open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.

• videoDashboards Are Dead — Sarah Simionescu, Composio
• videoWe Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI
• videoStop Fine-Tuning to Fix Retrieval Problems — Anant Srivastava
• videoAn Interaction Is All You Need — Ivan Leo, Google DeepMind

• videoLocal Models: Trust, Control, Optimization — Carter Abdallah, NVIDIAAI Engineer
• videoLarge clusters for small models — Daniel Svonava, SuperlinkedAI Engineer
• videoWhat's New in Inference Engineering — Philip Kiely, BasetenAI Engineer

Checking sign-in…

Loading comments…