Reiner Pope – The math behind how LLMs are trained and served

Serving several AI requests together can reduce the cost of each response because they share part of the work of reading the model's parameters from memory. Reiner Pope, CEO of the chip startup MatX, explains this batching tradeoff in his lecture on model serving. Smaller batches can produce faster responses, but distribute that shared cost across fewer requests. Larger batches improve the economics until other work becomes limiting: each request still requires its own computation and processing of its conversation context. This explains both why a faster service can cost more and why accepting ever-slower responses cannot make generation arbitrarily cheap.

“if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together”

Reiner Pope

“Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem.”

Reiner Pope

“I mean, you are incentivized to price close to your costs because otherwise someone could scoop you.”

Reiner Pope

“This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.”

Reiner Pope

“The empirical result is that the context lengths haven’t been increasing that much. I think it’s because there is no solution to the memory wall here.”

Reiner Pope