Applying Transformer Inference Arithmetic to Hopper and Blackwell GPUs

wafer_ai is publishing a series on AI performance engineering, and in part 6 it applies Carol Chen's transformer inference arithmetic to Nvidia's H200, B200 and B300 GPUs in a long post. Chen, who writes as kipply, built an approximate cost model for running a language model from three quantities: the work each token requires, the bytes the GPU has to move, and the communication between GPUs. Her examples used older A100 GPUs. wafer_ai pairs her model with Nvidia's Hopper and Blackwell documentation and turns it into a tuning method: predict which resource limits a given setup, change the serving or kernel configuration that relieves it, and check the result against the workload's latency target.

The clearest example is the difference between the two phases of serving a request. In prefill, the model processes the prompt's tokens together, so each weight it loads from memory is reused across many tokens. In decode, which produces the output, small batches offer much less reuse, and the GPU can spend more time loading weights than computing with them. Batching more requests raises arithmetic intensity, the amount of computation done for each byte of weights loaded, which can raise throughput. Larger batches also need more key-value cache, the keys and values retained for earlier tokens that decoding reads at every step, and they can lengthen the time between output tokens. The post therefore treats batch size as something to tune against the latency target with realistic prompt and output lengths, and says to reject throughput gains that break that target.

The same accounting runs through the rest of the advice. Memory capacity decides which configurations fit, and the three GPUs carry different amounts each, but the post keeps capacity and bandwidth separate: a long context can fit in memory while reading its cached keys and values at every step still dominates latency. Precision can change both memory traffic and compute throughput, with FP8 on Hopper and native NVFP4 on B200 and B300, and the post says to count the scale metadata, confirm which kernels actually run, and test the quantized model on the tasks it must perform. Splitting a model across more GPUs with tensor parallelism reduces each GPU's computation while making communication a larger share of latency.

The post's text offers a method, not measurements of its own on the newer GPUs. Chen checked her estimates against FasterTransformer measurements, and wafer_ai asks readers to do the equivalent: compare predictions with measured kernel durations and achieved memory bandwidth, and record time to first token, inter-token latency and output-token throughput on the same model and a representative workload.