Vibeleaderboard
← All Intel
Intel / video

Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

Source
youtube.com
Author
Latent Space
Date

Context

In a Latent Space interview, Philip Kiely of Baseten, a company that hosts AI models, walks through what happens to a 200,000- request (tokens are the word fragments a model reads). First, routing checks whether part of the input was already processed. If so, it goes to a server holding that saved work, skipping repeated effort. Otherwise a group of GPUs does prefill, reading the whole input to produce the first token, and on some models a separate group handles decode, generating the rest.

Ali Taha explains : a small attached model guesses a few next tokens quickly, and the large model checks them in one pass, accepting or rejecting them. A draft model trained on one customer's kind of text, such as one book series or coding, is accepted more often. Taha says that kind of tuning is not possible on a shared endpoint, which cannot know what text it will serve. This is the speakers' description of their own system.

Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…