Vibeleaderboard
← All Intel
Intel / video

Jeff Dean: The 1% Rule for Building in AI

Source
youtube.com
Author
Y Combinator
Date

Context

Google's Jeff Dean argued in a Y Combinator conversation that energy costs shape how AI systems are built. Agreeing with the 's figure, he said moving data from memory into the processor costs about a thousand times the energy of computing on it. That is why accelerators batch many examples or together: they share one data transfer, so the cost per example falls in proportion to batch size, though batching works against low .

From that he predicted specialized hardware with lower energy use and latency than general-purpose GPUs or TPUs. He tied this to the TPU: a 2013 back-of-envelope estimate, as the host described it, showed a few minutes of daily speech recognition per user would force Google to double its fleet, and the chip built for low-precision linear algebra was, he said, 30 to 80 times more energy efficient than contemporary CPUs and GPUs. These are recollections and a forecast.

Terms in this piece · Glossary
  • Host — The computer on which a process runs.
  • Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
  • Latency — The time between asking for a result and having it, including the model's generation time, tool calls, retries and any queueing.
  • Inference — Using a trained model to produce an output from a supplied input.
Read the source www.youtube.com
More from Y Combinator
Recommended reads
Comments

Checking sign-in…

Loading comments…