inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Why it matters
Mamba-3 offers a state-space alternative to Transformers with faster decode-time inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → and open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition →, making it worth evaluating for latency-sensitive or long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → serving where attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition →-based decoding is the bottleneck.
Key quotes
“Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding.”
Together AI
“Mamba-2 made the bet that training efficiency was the largest bottleneck for state space models (SSMs), and thus simplified the underlying SSM mechanism to deliver 2−8× faster training compared to its predecessor, leading to wider adoption.”
Together AI
“The same fixed state size that enables efficient computation forces the model to compress all past information into one representation, the exact opposite of a Transformer, which stores all past information through a continuously growing state (the KV cache) --- a fundamental difference.”
Together AI
“Current linear models have been designed to use lots of GPU tensor cores (one of the main contributions of Mamba-2) for fast training, but during decoding, each timestep requires so little compute that the hardware remains cold most of the time.”
Together AI
“When comparing models at the 1.5B scale, Mamba-3 (SISO variant) achieves the fastest prefill + decode latency across all sequence lengths, outperforming Mamba-2, Gated DeltaNet, and even the Transformer with its highly optimized vLLM ecosystem.”
