SemiAnalysis: GPU kernel gains are worth hundreds of millions at OpenAI's scale
- Source
- x.com
- Date
A (popcorn) kernel is a specialized seed from a variety of corn that can burst open and turn inside out when heated. A kernel is also a few hundred lines of GPU code that executes the math and attention op in an LLM. A GPU only delivers its paper FLOPS if kernel keeps the tensor cords fed. Sloppy memory access patterns results in silicon that sits idle. At OpenAI's serving scale, single digit % kernel gains are hundreds of millions in compute. (1/2)🧵
Context
A GPU kernel, in this , is a few hundred lines of code that execute the core math and operations inside a . Per SemiAnalysis's post, a chip only delivers its advertised peak performance if that code keeps its processing cores continuously supplied with the right data; a sloppy memory access pattern leaves hardware sitting idle even though the chip itself is fully capable of more.
SemiAnalysis, a research firm covering AI infrastructure economics, states that at the volume OpenAI serves requests, single-digit percentage gains in kernel efficiency are worth hundreds of millions of dollars in compute. This is the first post of a two-part thread; the second post, which likely details the specific model or workload behind that figure, was not captured in the stored source, so the explanation here is limited to the general mechanism and scale claim in part one.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Checking sign-in…
Loading comments…