Intel / article

How to improve AI energy efficiency by 1000x

Source: unconv.ai
Author: Peter McMahon
Date: Published May 7, 2026

Context

Unconventional AI's proposed thousandfold energy improvement requires redesigning nearly the entire inference system. Its technical argument defines success as producing comparable-quality text or images with less total energy, including the cost of storing and moving model data.

The author's coarse estimate for a model with 100 billion active parameters shows why memory matters: about 0.007 joules per token for the arithmetic alone, against about 3.9 joules per token just to read those parameters from off-chip memory, assuming no reuse of the values read. A simple accounting constraint adds to the difficulty: if an unchanged component consumes 1% of the original energy, eliminating every other cost still yields at most a hundredfold improvement. The company therefore proposes jointly changing models, memory, and physical layout. The thousandfold figure describes the target of that program, not a reported end-to-end measurement.

Key takeaways · AI-distilled

Back-of-envelope math: reading a 100B-parameter model's weights from off-chip HBM costs about 3.9 Joules/token versus about 0.2 J/token from on-chip SRAM and just 0.007 J/token for the arithmetic itself, meaning memory access is the real target for a 1000x gain.
Even moving all parameter reads from off-chip HBM to on-chip SRAM buys at best a 10x advantage under best-case assumptions; the measured real-world gain for models is closer to 1.5x, so on-chip memory alone can't reach 1000x.
At long lengths the rivals the parameters in memory traffic: a 32k-token context can require reading about 50GB (roughly 2 J/token), and a 1M-token context could exceed 1TB, on the order of 100 J/token just for cache reads.
Amdahl's Law bounds any hardware win: optimizing a part of the system that starts at 99% of the energy budget caps the best possible system-level improvement at 100x even if that part is driven to zero, so overhead beyond 0.1% of the budget defeats a 1000x goal.
The piece cites specific levers already tried elsewhere, including hard-wiring model parameters into silicon (done by Taalas) and avoiding /KV-cache loading via state-space models like Mamba, arguing 1000x likely comes from stacking several such wins.

Terms in this piece · Glossary

transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Read the source unconv.ai