Vibeleaderboard
← All Intel
Intel / video

How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

Source
youtube.com
Author
Dwarkesh Patel
Date

Context

Reiner Pope, CEO of the chip startup MatX and a former Google TPU architect, explains in a blackboard lecture hosted by Dwarkesh Patel why paying more can buy faster output: the main lever is batch size, the number of users served in one pass. Patel discloses being an angel investor in MatX.

In Pope's model, one decoding step, which produces one new , takes at least as long as the larger of its compute time and its memory time. Each step must read all the model's weights from memory, so latency has a floor however few users share the pass. Cost per token is step time divided by batch size, so a batch of one pays for the whole weight read alone and cost falls as more users share it. The , the stored internal record of earlier text that each request must re-read, adds a memory cost that grows with . Pope presents this as a simplified lower bound, not an exact prediction.

Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Read the source www.youtube.com
More from Dwarkesh Patel
Recommended reads
Comments

Checking sign-in…

Loading comments…