Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
In a Latent Space interview, Philip Kiely of Baseten, a company that hosts AI models, walks through what happens to a 200,000- request (tokens are the word fragments a model reads). First, routing checks whether part of the input was already processed. If so, it goes to a server holding that saved work, skipping repeated effort. Otherwise a group of GPUs does prefill, reading the whole input to produce the first token, and on some models a separate group handles decode, generating the rest.
Ali Taha explains : a small attached model guesses a few next tokens quickly, and the large model checks them in one pass, accepting or rejecting them. A draft model trained on one customer's kind of text, such as one book series or coding, is accepted more often. Taha says that kind of tuning is not possible on a shared endpoint, which cannot know what text it will serve. This is the speakers' description of their own system.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
videoThe Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTOLatent Space
videoThe AI Frontier: from open weights to open research — Eiso Kant, Poolside AILatent Space
videoDevin’s 80% Moment: Background Agents, 7x PRs, & End of Hand-Held Coding — Walden Yan & Cole MurrayLatent Space
Checking sign-in…
Loading comments…



