Pushing software engineering limits with “napkin math”
Source
Gergely Orosz
Author
Gergely Orosz
Date
Key takeaways · AI-distilled
Napkin math means computing the theoretical floor first, the raw cost of moving those bytes or reading that data, then comparing it to what the system actually does. The gap tells you whether you are paying for physics or for waste.
That gap is what created turbopuffer. Early ChatGPT context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → windows were small, so retrieval was mandatory, and rough first-principles estimates showed existing search products cost far more than the underlying operations required.
A competitive-programming lesson that generalizes: if you are writing code to handle a specific edge case, you probably have the wrong solution. Two edge cases means two more you have not found; the simpler general solution absorbs them all.
When you are fixated on an approach that is not working, delete the program and start on a fresh sheet. The time already spent is smaller than the hole you keep digging.
A cheap way to close a knowledge gap without a CS degree: note every term you did not recognize at work that day, and read up on each one that evening. Repeat for a year.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Learn to use 'napkin math' — rough first-principles estimates of compute, I/O, and memory limits — to quickly spot why AI-era systems run slow or cost too much, illustrated through the origins of vector search infra like turbopuffer.
Key quotes
“But it should take 10 milliseconds if you do the napkin math.”
Simon Eskildsen
“For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents.”
