Vibeleaderboard
← All Intel
Intel / video

2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]

Source
youtube.com
Author
Latent Space
Date

Context

, the step where a compares every token (a word-sized piece of text) with every other token, costs computation that grows with the square of input length. In a NeurIPS 2024 talk, Dan of Together AI said this matters more as models spend extra tokens thinking, and framed the goal as an operator closer to n log n.

He described two routes. Linear attention drops the softmax step so the square-cost comparison never happens, but around 2020 it gave worse quality and ran slower in practice. space models (S4, 2022) reused a simple recurrent update from signal processing, rewritten as a convolution that a fast Fourier transform computes in n log n time. A language quality gap remained, which he tied to picking details out of the model's summary state. He credited gating and Mamba's data-dependent matrices for lifting quality. This is one researcher's overview, not a test against Transformers.

Terms in this piece · Glossary
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
  • State — Information describing a system at a particular moment, such as its current text, settings or saved records.
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…