That's why LLM inference engines put a lot of thought into optimizing the decoding step, which is the bottleneck . You might be familiar with vLLM, SGLang or TensorRT. All these frameworks increase the arithmetic intensity of LLM inference with smart engineering hacks. When the inference algorithm itself limits parallelism for a single query, the most obvious fix is to serve multiple queries in parallel. That's known as batching. An inference server, which is a software component that exposes an API, receives requests with text prompts. It groups them into batches and then ships them off to the GPU. Decoding still happens iteratively one step at a time, but a single forward pass through the model is now extending all responses by an output token. The good news is that it's very easy to modify the computational graph to support batching. Most operations multiply a weight matrix W by an activation x. Without batching, x is a vector. With batching, x is a matrix with one column per query. Now storing a larger x in the on-chip memory does put some pressure on its capacity. But crucially, the really expensive part, which is streaming the model weights from the HBM, still only happens once. From an arithmetic intensity point of view, the numerator increases while the denominator stays constant. This pulls the inference algorithm upwards on the roofline.