Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Source
youtube.com
Author
Dwarkesh Patel
Date
Why it matters
Hearing the people behind the transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition →, mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → and Gemini discuss how large systems and models evolved gives context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → for where model and infrastructure design is heading.
Terms in this piece · Glossary
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
