A 1T-parameter model breaks 1,000 tokens per second on one standard 8-GPU node
- Source
- x.com
- Date
🚀 1,000+ TOKENS/S ON A 1T MODEL! 🚀 We are thrilled to release Xiaomi MiMo-V2.5-Pro-UltraSpeed in collaboration with @TileRT_AI , breaking the 1,000 tokens/s output speed on a 1 Trillion parameter model for the FIRST TIME! Not wafer-scale integration like Cerebras. Not pure on-chip SRAM chips like Groq. We achieve 1,000 tps on a 1T MoE model using just a SINGLE, STANDARD 8-GPGPU NODE. Read the full technical deep dive：https://t.co/MX0kjHKdKi Want to experience the future of real-time AI? 👉 Apply for UltraSpeed now: https://t.co/aeWAxyhwVk ⏳ Limited-Time Access: Application-based · Jun 8 – Jun 23 (PDT) 💬 Chat Experience: Completely FREE for a limited time — try the blazing-fast web chat now. ⚡ UltraSpeed API: Just 3x the price for a ~10x boost in output experience. 🤝 Enterprise & Large-Scale Needs: business-mimo@xiaomi.com
🔓 And the best part — we're open-sourcing it. 1,000+ tps on a 1T model wasn't a single breakthrough — it's deep model × system co-design between the MiMo and TileRT teams, all on general-purpose GPUs (no Cerebras-style wafer-scale, no Groq-style SRAM ASICs). On the model side: FP4 quantization (smaller footprint, less memory traffic) + DFlash, our block-masked parallel speculative decoding that accepts far more tokens per verification. On the system side, TileRT tailors its compiler & kernels to exactly these techniques. The result: a 1T model breaking 1,000 tps on a single, standard 8-GPU node. 🤗 Open weights (FP4 + DFlash checkpoint): https://t.co/jYQsgeruMg
Context
Xiaomi's MiMo lab, working with TileRT, reports its MiMo-V2.5-Pro-UltraSpeed variant is the first 1-trillion-parameter model to exceed 1,000 output per second, running on one standard 8-GPU node rather than Cerebras-style wafer-scale silicon or Groq-style SRAM chips built for that kind of throughput.
The company attributes the result to two combined techniques: FP4 , which shrinks the model's memory footprint and the data moved per token, and DFlash, a block-masked method that accepts more draft tokens per verification pass than typical approaches. Xiaomi says TileRT tuned its compiler and kernels specifically around these two techniques, and both the FP4 weights and the DFlash checkpoint were released openly.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Checking sign-in…
Loading comments…