However, a single model transfer serves a single forward pass, and, in an auto-regressive LLM, a single output token. So an average size response of 1000 tokens requires 1000 full model streams. With this perspective, the latency we saw earlier is less surprising. An 8 billion model roughly takes 16 gigabytes of storage. With a memory bandwidth of 3.2 terabytes per second, we can afford to make exactly 200 transfers in a second. Assuming a negligible GPU runtime, that's a throughput of 200 tokens per second. This shows that model inference is heavily bottlenecked by memory bandwidth. My initial rough math showed that the GPU processor at its peak could handle hundreds of times more operations than this. Yet it's sitting idle, waiting for parameters to arrive. Now there are two things we could do about it.