🚀 Meet HySparse: Our new breakthrough in long-context LLM efficiency!
- Source
- x.com
- Date
🚀 Meet HySparse: Our new breakthrough in long-context LLM efficiency! We’re excited to share HySparse (Hybrid Sparse Attention)—a hybrid model architecture that interleaves each full attention layer with multiple sparse attention layers, where the sparse layers strategically derive important token selection and KV caches from the preceding full layer！ 📖 Paper link:
🔍 What makes HySparse different? HySparse with Oracle Token Selection and KV Cache Sharing resolves two fundamental limitations of prior sparse attention methods: ⚠️ Inaccurate Proxy Selection: Conventional methods rely on approximate "proxies" or additional modules to guess token importance. HySparse uses the full attention layer as a precise oracle to identify important tokens. No more proxy guessing which tokens matter. 💾 High Memory Overhead: Existing dynamic sparse attention approaches reduce computation but fails to reduce KV cache storage. Instead of creating new memory for every layer, HySparse enables sparse attention layers to reuse the full attention KV cache.
📈 Proven Performance HySparse demonstrates in experiments that an 80B MoE model requires only 5 out of 49 layers to utilize full attention, enabling a nearly 10x reduction in KV cache storage. Despite this efficiency, the model improves performance over full attention baselines on general and long-context benchmarks.
Context
Sparse normally cuts compute by skipping less-important tokens, but it still needs a full in memory because most methods guess which tokens matter using a separate proxy module. Xiaomi's MiMo lab describes HySparse, an architecture that interleaves each full-attention layer with several sparse-attention layers and lets those sparse layers reuse the token selection and KV cache the preceding full-attention layer already computed, removing the separate proxy.
In the team's own reported experiments, an 80-billion-parameter model needed full attention in only 5 of its 49 layers, which the paper credits with close to a 10x reduction in KV cache memory while still scoring ahead of full-attention baselines on general and long- benchmarks. These are the developers' measured results on their own model and benchmark suite, not an independent replication.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…