wafer

A thread from Wafer's performance engineering series mapping NVIDIA's CUDA Programming Guide to LLM serving: kernel tile tuning, overlapping data movement, growing KV caches via virtual memory, and reducing launch overhead with CUDA Graphs.

Wafer updates its open source GPU performance engineering guide, adding Stream-K, a Blackwell tcgen05 matmul walkthrough, batch-invariant kernels for deterministic inference, and Triton Linear Layouts. The guide runs from one inference request to kernels, serving engines, and distributed systems.

A thread summarizing the Etalon framework for evaluating LLM inference: how chunked prefill and speculative decoding shape token timing, why averages mask pauses, and how to set first-token deadlines.

A Wafer thread condensing NVIDIA's CUDA C++ Best Practices Guide for AI performance engineers: profiling with Amdahl's law, bandwidth measurement, memory coalescing, shared-memory tiling, occupancy tradeoffs and numerical validation. Links a curated GPU performance engineering resource list.

A performance-engineering series builds on Carol Chen's transformer inference cost model, applying it to H200, B200 and B300 GPUs to reason about prefill/decode tradeoffs, batching, and memory budgeting for serving.

Wafer breaks down how 'Programming Massively Parallel Processors' applies to AI performance engineering, connecting CUDA execution, memory coalescing, tiled matrix multiplication, and occupancy tuning to real GPU kernel bottleneck diagnosis.

Wafer reports its GLM-5.2 endpoint beat a Cerebras-served Gemma 4 31B on latency (379ms vs. 674ms average) in Y Combinator's AI Office Hours product, driving 2.5 extra minutes of user engagement per session.

A thread distilling the PaLM TPU v4 inference scaling paper into practical checks: separating prefill and decode bottlenecks, sweeping batch and context length, and choosing between 1D and 2D partitioning and weight-stationary layouts.