CUDA Best Practices Distilled for AI Kernel Tuning

Wafer is posting the resources in its curated AI Performance Engineering list one at a time, and part 12 of the series covers NVIDIA's CUDA C++ Best Practices Guide. The guide is a general manual for making GPU programs fast, not an AI document. Wafer's contribution is to read it as a decision aid: when a training step or inference request is slow, it helps decide whether to change a tensor's memory layout, move less data, or tune how a kernel uses the chip's resources. An earlier part did the same for NVIDIA's CUDA Programming Guide.

The guide starts from measurement. Its cycle, which NVIDIA calls Assess, Parallelize, Optimize, Deploy, begins by profiling to find where time goes, then uses Amdahl's law, which says the speedup from accelerating one part of a program is capped by that part's share of total runtime, to estimate what a fix could be worth. Wafer adds that for inference this profiling should use realistic batch sizes and sequence lengths. The guide times GPU work with CUDA events, timestamps recorded on the GPU itself, and makes effective bandwidth, bytes read and written divided by time, a high-priority metric; comparing the memory traffic a kernel requests with what the hardware actually moves shows how much bandwidth a poor access pattern wastes. Because the link between CPU and GPU is far slower than GPU memory, the guide says intermediate data should stay on the device even if some kernels there are no faster than on the CPU.

The worked matrix multiplications are where these ideas become concrete. A GPU runs threads in groups of 32 called warps, and when neighboring threads read neighboring addresses the hardware merges their reads into a few transactions, which is called coalescing. In the guide's example on an NVIDIA Tesla V100, a simple multiply reaches 119.9 GB/s of effective bandwidth, staging a tile of one matrix in shared memory, the fast on-chip memory threads in a block share, raises it to 144.4 GB/s, and also loading the second matrix's tile once lifts it to 195.5 GB/s. A transposed variant shows the failure case: strided reads drop it to 12.8 GB/s, a shared transposed tile restores 140.2 GB/s, and the remaining loss comes from bank conflicts, where shared memory's 32 banks serialize requests that land on the same bank. Padding the tile with one extra column removes them and brings it to 199.4 GB/s. That one-line layout change is what Wafer points to as something to try inside tensor kernels.

The last two lessons are about tradeoffs and correctness. Occupancy is the share of the chip's possible active warps actually running, and giving each thread more registers can avoid spilling to slower memory while letting fewer blocks run at once; the guide says higher occupancy does not always mean higher performance, so Wafer's advice is to time several configurations. Faster math intrinsics such as __expf trade accuracy for speed, and the guide calls the compiler flag that applies them everywhere an aggressive optimization. Since floating-point results may not match bit for bit after such changes, it recommends comparing against known-good reference outputs within a tolerance, which Wafer applies to custom softmax and normalization kernels. Its closing point is that a kernel win only counts if the full training or inference workload gets faster and stays accurate.