A (popcorn) kernel is a specialized seed from a variety of corn that can burst open and turn inside out when heated. A kernel is also a few hundred lines of GPU code that executes the math and attention op in an LLM. A GPU only delivers its paper FLOPS if kernel keeps the tensor cords fed. Sloppy memory access patterns results in silicon that sits idle. At OpenAI's serving scale, single digit % kernel gains are hundreds of millions in compute. (1/2)🧵
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Illustrates why kernel-level optimization work is disproportionately valuable at hyperscale inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →: small, unglamorous efficiency gains compound into massive cost savings, a lens worth applying to any team's own serving costs.
