Vibeleaderboard
← All Intel
Intel / post

Mapping the CUDA Programming Guide to LLM inference bottlenecks

Source
x.com
Date
wafer_ai@wafer_ai

we launched the most comprehensive ai performance engineering repo in the world and we're posting every single resource follow and save to keep up with the series. links in thread 🧵 part 10: CUDA Programming Guide making an LLM faster means understanding where execution is waiting. an attention kernel can wait for its next tile of data, while the serving runtime can leave the GPU waiting for the next launch. NVIDIA's CUDA Programming Guide explains the mechanisms behind those waits. for a modern AI performance engineer working in PyTorch, Triton, or CUDA, it helps with: - tuning attention, matmul, and normalization kernels. the SIMT and CUDA Tile chapters connect layouts and data reuse to GPU execution. the resource guidance explains why larger tiles or more fusion can introduce register spills. - overlapping data movement with computation. asynchronous copies, TMA, and producer/consumer pipelines explain how supported GPUs can load the next tile while processing the current one, with the synchronization required to reuse buffers safely. - managing growing KV caches. the virtual memory chapter explicitly connects its allocation model to LLM serving: reserve an address…

Read the full post on X

Context

Wafer argues in part 10 of its X series on AI performance engineering that making a faster starts with finding where execution is waiting, and that NVIDIA's CUDA Programming Guide explains why those waits happen. CUDA is NVIDIA's platform for programming its GPUs. The post gives two kinds of waiting. An kernel, the small GPU that computes a model's attention step, can stall until its next tile of data arrives. The serving software can also leave the GPU idle while it sets up the next kernel launch.

The post then matches parts of the guide to specific inference problems. The clearest example is the , the stored attention data a model keeps for text it has already processed, which keeps growing during a request. According to the post, the guide's virtual memory chapter ties its allocation model directly to LLM serving: a program reserves a range of addresses up front, then maps more physical memory into that range as the cache grows, so existing data never has to be moved. The other pairings work the same way. The post says asynchronous copies let supported GPUs load the next tile while computing on the current one, and CUDA Graphs cut the repeated CPU work of launching kernels. It also says memory synchronization domains can, on supported hardware, stop a compute kernel from waiting on unrelated writes by NCCL, NVIDIA's library for moving data between GPUs.

These are Wafer's descriptions of the guide, and the post reports no measured speedups. Its practical point is about diagnosis. When a training step or inference request slows down, knowing these mechanisms helps a developer pick the next experiment: change the kernel, move less data, adjust memory allocation or remove a wait that isn't needed. The series works through Wafer's open AI performance engineering resource list, which Intel covered separately.

Terms in this piece · Glossary
  • Large language model — A model trained on large amounts of data to process and generate language.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • Program — A set of instructions stored so that a computer can run it.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
More from wafer_ai
Recommended reads
Comments

Checking sign-in…

Loading comments…