Intel / article

Mamba-3

Source: together.ai
Author: Together AI
Date: Published Mar 17, 2026

Photo by National Cancer Institute on Unsplash

Context

Mamba-3 tries to get more useful computation from a fixed-size memory of previous text. Such state-space models compress earlier into a state that is updated as text arrives. Together AI's research account explains that earlier designs spent much of generation moving data while leaving arithmetic capacity unused.

Mamba-3 makes the state update richer, including a variant that processes multiple input and output channels. The authors report improved language-modeling results with roughly comparable decoding latency, because the extra arithmetic can use otherwise idle compute. Training costs increase for that variant, and pure state-space models still trail the baseline on retrieval tasks. Its fixed memory therefore offers an efficiency tradeoff whose value depends on the task.

Terms in this piece · Glossary

context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Key quotes

“Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding.”
Together AI

“Mamba-2 made the bet that training efficiency was the largest bottleneck for state space models (SSMs), and thus simplified the underlying SSM mechanism to deliver 2−8× faster training compared to its predecessor, leading to wider adoption.”
Together AI

“The same fixed state size that enables efficient computation forces the model to compress all past information into one representation, the exact opposite of a Transformer, which stores all past information through a continuously growing state (the KV cache) --- a fundamental difference.”
Together AI

“Current linear models have been designed to use lots of GPU tensor cores (one of the main contributions of Mamba-2) for fast training, but during decoding, each timestep requires so little compute that the hardware remains cold most of the time.”
Together AI

“When comparing models at the 1.5B scale, Mamba-3 (SISO variant) achieves the fastest prefill + decode latency across all sequence lengths, outperforming Mamba-2, Gated DeltaNet, and even the Transformer with its highly optimized vLLM ecosystem.”
Together AI

Read the source www.together.ai