we launched the most comprehensive ai performance engineering repo in the world follow and save to keep up with the seri
- Source
- x.com
- Date
we launched the most comprehensive ai performance engineering repo in the world follow and save to keep up with the series. links in thread 🧵 part 6: Transformer Inference Arithmetic Carol Chen (kipply) builds an approximate inference cost model around the work each token requires, the bytes the GPU moves, and communication between GPUs. it's a useful starting point for investigating latency and throughput on H200s, B200s, and B300s. Chen's examples use A100 GPUs. the applications below combine that model with NVIDIA's Hopper and Blackwell documentation: - prefill and decode expose different amounts of token parallelism. prefill processes prompt tokens together, giving projection and feed-forward matrix multiplies more weight reuse. small-batch decode has less reuse and can spend more time loading weights than computing with them. profile each phase. - batching can raise arithmetic intensity in projection and feed-forward matrix multiplies: more computation for each byte of weights loaded. that can improve throughput, but larger batches can increase key-value (KV) cache requirements and the time between output tokens. tune batch size against the latency target, using…
Context
wafer_ai's post is one entry in a series applying Carol Chen's 2022 cost model for , built around the work each output token requires, the bytes a GPU has to move, and communication between GPUs, to current Nvidia hardware including H200, B200, and B300 GPUs.
One of the model's central points, echoed in wafer_ai's post, is that generating tokens one at a time, called decoding, is usually limited by how fast a GPU can read its stored weights and cached from memory, not by how fast it can do the arithmetic. Because of that, processing more requests together in one batch lets the GPU do more computation for every byte of weight and cache data it has already loaded, which is why batching raises throughput. The tradeoff is that larger batches also mean each request's , the running memory of its own conversation so far, has to share GPU memory with everyone else's, and can increase the time between output tokens for any single request; the post frames choosing a batch size as a matter of testing against a target latency, using representative prompt and output lengths, rather than assuming bigger is always better.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Checking sign-in…
Loading comments…