Back-of-envelope math: reading a 100B-parameter model's weights from off-chip HBM costs about 3.9 Joules/token versus about 0.2 J/token from on-chip SRAM and just 0.007 J/token for the arithmetic itself, meaning memory access is the real target for a 1000x gain.
Even moving all parameter reads from off-chip HBM to on-chip SRAM buys at best a 10x advantage under best-case assumptions; the measured real-world gain for transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → models is closer to 1.5x, so on-chip memory alone can't reach 1000x.
At long context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → lengths the KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → rivals the parameters in memory traffic: a 32k-token context can require reading about 50GB (roughly 2 J/token), and a 1M-token context could exceed 1TB, on the order of 100 J/token just for cache reads.
Amdahl's Law bounds any hardware win: optimizing a part of the system that starts at 99% of the energy budget caps the best possible system-level improvement at 100x even if that part is driven to zero, so overhead beyond 0.1% of the budget defeats a 1000x goal.
The piece cites specific levers already tried elsewhere, including hard-wiring model parameters into silicon (done by Taalas) and avoiding attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition →/KV-cache loading via state-space models like Mamba, arguing 1000x likely comes from stacking several such wins.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
