Information Theory for Language Models: Jack Morris
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
Jack Morris, a PhD student at Cornell Tech, argues in a Latent Space interview that classical information theory misses what matters for language models. Shannon's measure counts a plain-English paragraph and an encrypted copy of it as holding the same information, though only the plain text is easy to use. Morris points to V-information, a 2020 proposal that measures how much information a given computation can extract, as one possible lens on questions such as why a pretrained beats a randomly initialized one.
Morris's capacity work follows that framing. models on random data and measuring memorization, Morris reports that memorization plateaus at a ceiling called model capacity, roughly 3.6 to 3.9 bits per parameter for 32-bit training. Morris and the both flag limits: memorizing is not generalizing, and better architectures, optimizers or measurements could raise the number, so it is a first measurement, not a constant.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- Host — The computer on which a process runs.
- Path — A description of a file or folder location, written using directory names and separators.
Checking sign-in…
Loading comments…






