Intel / post

we launched the most comprehensive ai performance engineering repo in the world follow and save to keep up with the seri

Key takeaways · AI-distilled

Building on Carol Chen's (kipply) arithmetic, Wafer notes prefill processes prompt tokens together for more weight reuse, while small-batch decode can spend more time loading weights than computing, so each phase should be profiled separately.
Larger batches raise arithmetic intensity in projection and feed-forward matmuls, but they also grow KV-cache requirements and time between output tokens, so the thread advises tuning batch size against a latency target with representative prompt and output lengths.
KV-cache capacity and bandwidth need separate accounting: with full , decode reads every retained key and value each step, so a long can fit in memory while its reads dominate latency. Under grouped-query attention, size by KV-head count.
More tensor-parallel shards cut per-GPU compute but make NVLink communication a larger share of latency. The thread recommends checking estimates against measured kernel durations and rejecting throughput gains that violate the latency target.

Terms in this piece · Glossary

transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Why it matters

Explains how to reason about prefill versus decode costs, batching tradeoffs, and memory budgets when sizing inference on H200, B200, and B300 GPUs.