Vibeleaderboard
← All Intel
Intel / video

Information Theory for Language Models: Jack Morris

Source
youtube.com
Author
Latent Space
Date

Context

Jack Morris, a PhD student at Cornell Tech, argues in a Latent Space interview that classical information theory misses what matters for language models. Shannon's measure counts a plain-English paragraph and an encrypted copy of it as holding the same information, though only the plain text is easy to use. Morris points to V-information, a 2020 proposal that measures how much information a given computation can extract, as one possible lens on questions such as why a pretrained beats a randomly initialized one.

Morris's capacity work follows that framing. models on random data and measuring memorization, Morris reports that memorization plateaus at a ceiling called model capacity, roughly 3.6 to 3.9 bits per parameter for 32-bit training. Morris and the both flag limits: memorizing is not generalizing, and better architectures, optimizers or measurements could raise the number, so it is a first measurement, not a constant.

Terms in this piece · Glossary
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Host — The computer on which a process runs.
  • Path — A description of a file or folder location, written using directory names and separators.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…