When implemented naively, batching comes with an opportunity cost. Some responses are longer than others. A binary yes or no question requires fewer iterations than an AI slop article, yet it takes up space in the batch until all instances are complete. That's why most inference engines today do continuous batching, also called iteration level scheduling. That means the batch is reconfigured after every iteration, so finalized instances are immediately swapped out for a new query. This way, there are no instances sitting idle in the batch and no bubbles in the pipeline. On the next iteration, all the members in the batch are updated with an extra output token and the cycle repeats. So batching involves some overhead, but it's worth it because it