That's why diffusion LLMs generally have higher arithmetic intensity and are more likely to be compute bound, falling to the right of the ridge point. How far exactly, directly depends on their context window size. Now if we only judge diffusion LLMs by the roofline model, they look ideal. They fully utilize the available GPU parallelism. But the roofline model is an incomplete picture because it doesn't capture productivity. We could game it by giving the processor busy work like repeatedly multiplying some matrices only to throw them away. In fact, if we look at the runtime measurements, early diffusion models like LLaDA were generally slower than traditional ones. And the larger their context window, the slower they become. So out of the box diffusion LLMs are not necessarily a silver bullet for speed. They just make a different trade off and ask for different kinds of optimizations.