The nuance comes from the fact that LLM inference is actually a two-staged procedure. Stage one processes the text prompt into real valued embedding vectors. This is called prefill. Stage two generates one token at a time conditioned on the previous ones. This is called decode. From a performance point of view, this distinction is meaningful because the two stages fall into different regions of the roofline. Decode is memory bound and prefill is compute bound. The root cause of this discrepancy is the fact that, during prefill, all the prompt tokens are readily available. And thanks to the design of the Transformer architecture, they can all be processed simultaneously. Let's assume the prompt has three tokens. With a single forward pass through the model, and therefore a single model stream, we manage to process all of them. So during prefill, the ratio of tokens to model streams is three to one. This arithmetic intensity grows linearly with the sequence length. So for typical prompt lengths of hundreds or thousands of tokens, prefill easily crosses the ridge point. In contrast, during the decode stage, we need three separate forward passes