← All IntelClip / EducationNapkin math: using first-principles estimates to catch bad benchmarks
From Fireside Chat: Simon Eskildsen on Building Turbopuffer · ≈16:04
“So napkin math was essentially just this table that I maintain on GitHub of how much bandwidth can you drive to DRAMM, what does a roundtrip to S3 cost, and how long does it take, how much bandwidth can you drive to an NVME SSD, how much bandwidth can you drive to an EBS volume?”
“I hate benchmarks so much because that's not a satisfying answer to me to me it's like this does not jive maybe my intuition database A that you're saying takes 10 seconds to do this should take 10 milliseconds if you do the napkin math right”
“Either there's a gap in my understanding, which is very likely, or you would benchmark the wrong thing.”
What’s in it
- Shares a public GitHub table of memorized hardware cost and bandwidth numbers
- Explains a flashcard method for memorizing DRAM, S3, NVMe, and EBS costs
- Shows how back-of-envelope math exposes flawed database benchmarks
Clip transcript
to do. And so I one of the projects I had while I was at Shopify was this napkin math project. Have you seen this >> napkin math? No. >> No. Um, so napkin math was essentially just this table that I maintain on GitHub of how much bandwidth can you drive to DRAMM, what does a roundtrip to S3 cost, and how long does it take, how much bandwidth can you drive to an NVME SSD, how much bandwidth can you drive to an EBS volume? Just a collection of probably there's probably like 50 of these numbers and then a RS script that generates them all. um what all these things cost? What do you like? What does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? 2 cents. What does a gigabyte of um this cost? 10 cents, right? What does it cost on spot? What does it cost on a three-year commit? Like I had just have a massive table and then create flash cards for almost every single cell. So I know all these numbers. And this was a project I started taking on at Shopify because I found myself in um this role a lot where I would go in and review a project, right? So some product team would be like okay we got to do we got to build this thing so we got to build this infrastructure to support the feature and a lot of the times they would say okay well we've gone and benchmarked it on database A but the benchmarks are not very good so we're going to go with database B and I hate benchmarks so much because that's not a satisfying answer to me to me it's like this does not jive maybe my intuition database A that you're saying takes 10 seconds to do this should take 10 milliseconds if you do the napkin math right if it's a search query right it's like okay you're searching for three terms there each term has this many documents that match it that's this many megabytes we inter intersect these many this many lists you have DRAM bandwidth on multiple cores of 100 gigabytes per second this should take 10 millisecond you tell me the benchmark takes 10 one of us is wrong. Either there's a gap in my understanding, which is very likely, or you would benchmark the wrong thing. And in some ways, some reasons, right, it's like, okay, you've done a benchmark. You don't didn't realize that your benchmark is doing a distributed query across a 100 different nodes. And so, of course, the P99 is going to be really, really high, right? Unless you've cut that off or or made some different set of trade-offs. So I just found myself in these discussions repeatedly where people were making infrastructure decisions based on poor
Comments
Sign in to comment.
Loading comments…