2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
, the step where a compares every token (a word-sized piece of text) with every other token, costs computation that grows with the square of input length. In a NeurIPS 2024 talk, Dan of Together AI said this matters more as models spend extra tokens thinking, and framed the goal as an operator closer to n log n.
He described two routes. Linear attention drops the softmax step so the square-cost comparison never happens, but around 2020 it gave worse quality and ran slower in practice. space models (S4, 2022) reused a simple recurrent update from signal processing, rewritten as a convolution that a fast Fourier transform computes in n log n time. A language quality gap remained, which he tied to picking details out of the model's summary state. He credited gating and Mamba's data-dependent matrices for lifting quality. This is one researcher's overview, not a test against Transformers.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- State — Information describing a system at a particular moment, such as its current text, settings or saved records.
- Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
Checking sign-in…
Loading comments…






