Jeff Dean: The 1% Rule for Building in AI
- Source
- youtube.com
- Author
- Y Combinator
- Date
Context
Google's Jeff Dean argued in a Y Combinator conversation that energy costs shape how AI systems are built. Agreeing with the 's figure, he said moving data from memory into the processor costs about a thousand times the energy of computing on it. That is why accelerators batch many examples or together: they share one data transfer, so the cost per example falls in proportion to batch size, though batching works against low .
From that he predicted specialized hardware with lower energy use and latency than general-purpose GPUs or TPUs. He tied this to the TPU: a 2013 back-of-envelope estimate, as the host described it, showed a few minutes of daily speech recognition per user would force Google to double its fleet, and the chip built for low-precision linear algebra was, he said, 30 to 80 times more energy efficient than contemporary CPUs and GPUs. These are recollections and a forecast.
- Host — The computer on which a process runs.
- Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
- Latency — The time between asking for a result and having it, including the model's generation time, tool calls, retries and any queueing.
- Inference — Using a trained model to produce an output from a supplied input.
Checking sign-in…
Loading comments…






