Measuring inference in tokens per megawatt
- Source
- x.com
- Date
A useful metric for evaluating LLM inference is tok/s/MW: how many tokens a system generates per second for each megawatt of all-in provisioned power. At one concurrent user, the B300 configuration below generates ~14 output tok/s per GPU on DeepSeek V4. The SemiAnalysis Datacenter Industry Model estimates 1.9 kW (0.0019 MW) of provisioned power per B300. 14 tok/s ÷ 0.0019 MW = 7,368 tok/s/MW That’s roughly 7,400 output tok/s per provisioned megawatt. (1/4)🧵

Using the fact that 1 MW = 1,000,000 J/s and 1 joule = 0.000239 food Calories, we can convert power efficiency into tokens per Calorie: 7,368 tok/s/MW = 0.007368 tok/J 0.007368 ÷ 0.000239 = 30.8 output tok/Calorie Given that an average human speaks at ~3.3 tok/s while the brain consumes ~20 W: 3.3 tok/s ÷ 20 J/s = 0.165 tok/J 0.165 ÷ 0.000239 = ~690 output tok/Calorie That makes human speech roughly 22× more energy-efficient than this concurrency 1 B300 configuration! (2/4)

Finally, a Big Mac contains roughly 580 calories which means: 🟠 B300: 30.8 × 580 = ~18,000 output tokens per Big Mac 🟠 Human: 690 × 580 = ~400,000 output tokens per Big Mac One Big Mac can produce a lot of output tokens! (3/4)

Unfortunately, the average human is much dumber than most frontier language models. So while we can speak to throughput per Big Mac, goodput might be a different story 😅 (4/4)

Context
SemiAnalysis proposes measuring generated text against the power capacity provisioned to serve it. In its worked example, a B300 configuration running DeepSeek V4 at one concurrent user produces about 14 output per GPU. Dividing by its estimated 1.9 kilowatts of provisioned power per GPU gives roughly 7,400 tokens per second per megawatt.
That denominator matters: it is an estimate of provisioned capacity, not a direct measurement of electricity consumed by an individual answer. The ratio describes throughput under a particular configuration and load. Comparisons using it need to retain those conditions rather than treating the number as a fixed energy cost for every generated .
- throughput — How much model output a provider can generate over time, commonly measured in output tokens per second.
- Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
- Inference — Using a trained model to produce an output from a supplied input.
article[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day paybackwww.latent.space
clipBandwidth math yields 200 tokens/secJulia Turc
articleSpaceX 10GW in 2027 – Why It’s Real, Will Drive $300B ARR for SpaceX, and Why Microsoft Will Be the Largest OfftakerJeremie Eliahou Ontiveros
Checking sign-in…
Loading comments…

