Jeff Dean: The 1% Rule for Building in AI

Google's Jeff Dean argued in a Y Combinator conversation that energy costs shape how AI systems are built. Agreeing with the host's figure, he said moving data from memory into the processor costs about a thousand times the energy of computing on it. That is why accelerators batch many examples or tokens together: they share one data transfer, so the cost per example falls in proportion to batch size, though batching works against low latency.

From that he predicted specialized inference hardware with lower energy use and latency than general-purpose GPUs or TPUs. He tied this to the TPU: a 2013 back-of-envelope estimate, as the host described it, showed a few minutes of daily speech recognition per user would force Google to double its fleet, and the chip built for low-precision linear algebra was, he said, 30 to 80 times more energy efficient than contemporary CPUs and GPUs. These are recollections and a forecast.