⚡ Faster than Fast. Designed for Agentic AI. Introducing Xiaomi MiMo-V2-Flash — our new open-source MoE model: 309B total params, 15B active. Blazing speed meets frontier performance. 🔥 Highlights: 🏗️ Hybrid Attention: 5:1 interleaved 128-window SWA + Global | 256K context 📈 Performance: ⚔️ Matches DeepSeek-V3.2 on general benchmarks — at a fraction of the latency 🏆 SWE-Bench Verified: 73.4% | SWE-Bench Multilingual: 71.7% — new SOTA for open-source models 🚀 Speed: 150 output tokens/s with Day-0 support from @lmsysorg🤝 🤗 Model: https://t.co/4Etm0yZKTL 📝 Blog Post: https://t.co/5zxmcDuB6o 📄 Technical Report: https://t.co/crac1YTLYl 🎨 AI Studio:

🏗️ Architecture Deep Dive MiMo-V2-Flash uses hybrid attention: Sliding Window + Global in a 5:1 ratio → 8 hybrid blocks (5 SWA layers → 1 Global) → 128-token sliding window → 256 experts, 8 active per token → Natively 32K trained, extended to 256K context support 💡 Key Findings Attention sink bias is essential. Without it, SWA performance tanks. With it, SWA matches or beats full global attention. Smaller windows win. 128 vs 512? Similar on short benchmarks. But after 256K extension, 128-token SWA pulls ahead while 512-token SWA collapses. 🤯 ⚡ MTP: Train Faster, Infer Faster 3 lightweight heads (0.33B each) double as draft models for speculative decoding. Result? 2.5× decode speedup.

🧪 Beyond arch innovation, MiMo-V2-Flash is cooked via a NEW post-training paradigm Multi-Teacher On-Policy Distillation (MOPD) ❌ Problem: Post-training's "see-saw" effect — improve math, break coding. Enhance reasoning, hurt safety. ✅ Solution: Train specialized expert teachers, then distill ALL of them into one student via on-policy RL with token-level KL rewards. 🔥 Results: → 1/50 compute vs. training specialists → On par with ALL teachers in their own domains → No capability trade-offs → Iterative: today's student → tomorrow's teacher 🧠 Why it works: Dense token-level supervision + on-policy sampling + multi-teacher guidance = no more see-saw.

🧠 Why we built MiMo-V2-Flash with Multi-Token Prediction (MTP) — and why we're open-sourcing it? MTP unlocks speed gains at both inference and training. Here's the breakdown: 1️⃣ Faster LLM Decoding LLM decoding is memory-bound — low arithmetic intensity kills throughput. Batch parallelism helps FFN but not attention (each request has its own KV cache). MTP generates draft tokens → main model verifies in parallel → token-level parallelism without extra KV cache I/O ⚡ With 3-layer MTP: → 2.8–3.6 accepted tokens per step → 2.0–2.6x actual speedup 2️⃣ Accelerating RL Training Rollout is the #1 bottleneck in RL. MTP tackles it head-on: → Scales token-level parallelism instead of batch size — enabling stable, efficient on-policy RL with small batches → Eliminates GPU idleness from long-tail stragglers when batch sizes shrink to ~1 🎯 Faster, more stable, more efficient RL. 💡 TL;DR: MTP accelerates inference + RL training while boosting GPU efficiency. A critical unlock for agentic models — so we're open-sourcing it. Try it. Build with us. 🚀