Reiner Pope – The math behind how LLMs are trained and served
Source
Dwarkesh Patel
Author
Dwarkesh Patel
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
You get a first-principles derivation of LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → training and serving economics from someone who has worked across chip design, compilers, and TPU architecture — including how to infer what frontier labs are doing from a handful of equations and public API prices.
Key quotes
“if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together”
Reiner Pope
“Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem.”
Reiner Pope
“I mean, you are incentivized to price close to your costs because otherwise someone could scoop you.”
Reiner Pope
“This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.”
Reiner Pope
“The empirical result is that the context lengths haven’t been increasing that much. I think it’s because there is no solution to the memory wall here.”
Reiner Pope
Chapters
00:00:00How batch size affects token cost and speed
00:32:09How MoE models are laid out across GPU racks
00:47:12How pipeline parallelism spreads model layers across racks
01:03:37Why Ilya said pipelining is not wise
01:18:59Models may be 100x over-trained beyond Chinchilla-optimal
01:33:02Deducing long context memory costs from API pricing
02:04:02Convergent evolution between neural nets and cryptography
