Mistral Large 4: When KV-Cache Savings Cost Agent Performance

Mistral AI's technical report on Mistral Large 4 explains why the company undid most of the memory savings it had built into the model (alphaXiv). Mistral Large 4 is a mixture-of-experts model with 1 trillion total parameters, 52 billion of them active for each token, designed for reasoning and long-horizon agent tasks. Mistral launched it as a public preview in October (launch post). The savings targeted the key-value (KV) cache, the stored attention information about earlier tokens that a model consults each time it generates the next one. In the original design, three of every four layers used sliding-window attention and could see only the previous 512 tokens. Cross-layer attention let some full-attention layers reuse the cache of an earlier layer, and YOCO let full-attention layers in the second half of the network reuse the cache produced by the first half. Together these shrank the KV cache to one sixteenth of a full-attention baseline, and early ablations suggested little cost on perplexity and common benchmarks.

The cost appeared only after fine-tuning. When the team ran short supervised fine-tuning passes on intermediate checkpoints, perplexity and knowledge benchmarks looked as expected, but agentic evaluations that require long-context reasoning came in lower than expected. These included SWE-bench, which asks a model to resolve real GitHub issues, and the browsing-agent benchmark BrowseComp. Under the same training-token budget, SWE-bench Verified scores fell when the team moved from full attention to sliding windows and fell further when cross-layer sharing was added. The report concludes that models with equal language-modeling loss, the standard measure of how well a model predicts text, can perform very differently on agent tasks, while knowledge-based evaluations did not show the same gap. Its ablations also suggest that longer training can reduce the penalty.

Rather than start over, Mistral performed what it calls model surgery on an existing checkpoint before the second phase of pretraining. Former sliding-window layers were allowed to attend over the full context, and layers that had borrowed another layer's cache got their own key and value weights, copied from the nearest layer that already had them. Two added techniques, LeRoPE and SSMax, let the model learn its positional frequencies and per-head query scaling. A larger-scale test found that keeping YOCO's cache sharing performed better, so the final model is almost, but not entirely, full attention. To bring serving costs back down, the team then adapted the model to DeepSeek Sparse Attention (DeepSeek's report). In that scheme a lightweight indexer selects the top 2,048 earlier tokens for each query to attend to. Mistral trained the indexers for about 15.1 billion tokens while the main model stayed frozen, then trained both together for another 198 billion.

The broader lesson concerns how labs choose architectures. Mistral writes that comparable language-modeling loss does not imply comparable performance on long-horizon tasks and that 'new, better proxies must be found', and says agentic evaluations are now informing design choices throughout training as it explores sparse, linear and hybrid attention for coming releases. The evidence comes from Mistral's own ablations on one model family. It shows that this combination of KV-cache compression hurt agent performance at a given training budget, not that every memory-saving design does. The efficiency work continued after the surgery: Mistral reports that quantizing the model's routed experts to NVFP4 produces a 606 GB checkpoint that matches the FP8 reference on its evaluations and can be served on four GB300 GPUs.