Trillion Token Context. No, Really — Anima Anandkumar & Benedikt Jenik, Accelerated Understanding
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
Anima Anandkumar and a co-founder of Accelerated Understanding say the company is one model across many kinds of physics simulation, the way language models replaced separate spell-check and translation systems. In a Latent Space interview they describe the bet that a shared model can serve areas such as fluids, energy and aerospace because they share underlying principles.
Their specific claim concerns scale. Physical data has three space dimensions plus time, and they say they keep that structure instead of compressing it, so inputs reach into the trillions of values. They report training with trillion-length inputs and outputs, running at five trillion, and training models up to a trillion parameters. They also say a model of fixed size trained on several physics areas beat separate models given the same total parameters, which they take as evidence of shared learning.
All of this is company-reported. They decline to give architecture details, saying only that neural operators (models designed to work at flexible resolution) form the basis. The excerpt gives no independent results, so the claims remain unverified.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- Inference — Using a trained model to produce an output from a supplied input.
- Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
- Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
Checking sign-in…
Loading comments…






