Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- Source
- youtube.com
- Author
- Dwarkesh Patel
- Date
Context
Noam Shazeer and Jeff Dean, two of Gemini's three co-leads at Google according to host Dwarkesh Patel, explained in a podcast interview why AI hardware moved toward low-precision arithmetic. Shazeer said arithmetic is now very cheap and moving data comparatively expensive, and neural networks are built from matrix multiplications that do many operations per byte moved. Dean said Google's TPUs are reduced-precision linear algebra machines, built to fill chip area with arithmetic units.
They called it co-design. One of them said the first TPU was built around eight-bit integers before the team was sure models could be for serving, and that has since moved to INT4 or FP4, with some people using two bits or one. The risk is that algorithm designers resist low precision while chip designers wait to be asked, so someone must see the whole picture and show a model can be, say, three times faster. These are firsthand accounts of Google's own hardware path.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Checking sign-in…
Loading comments…