Remember, auto regressive models make predictions based on the past. So to compute the next token, we need to cache activations from the previously generated tokens. This is called the KV cache. K stands for keys, V stands for values. These are activations specific to the Transformer architecture. The KV cache holds an entry for every previously generated token in the batch. Because of its size, it must be stored in the HBM. During a forward pass, the relevant keys and values are streamed all the way to the processor, alongside the model weights. But as the batch grows, so does the KV cache. While the goal of batching is to ease memory bandwidth pressure, its gains are partially offset by the KV cache. Because of this, we often cannot scale batch sizes enough to keep the processor fully busy. Now there are further optimizations to ensure memory is used effectively and accommodates for as much data as possible. For instance, the vLLM inference engine came up with paged attention, a mechanism inspired by traditional operating systems to reduce memory fragmentation. But ultimately, memory capacity and bandwidth are still finite and so is the maximum batch size. We are reaching a point where system level engineering optimizations have