Vibeleaderboard
← All Intel
Intel / post

Fused MoE Pallas kernel halves Ling-2.6-1T prefill latency on TPU v7x

Source
x.com
Date
Ant Ling@AntLingAGI

It has been a privilege to collaborate so closely with the SGLang team @lmsysorg on optimizing Ling-2.6-1T. 🥳 The resulting performance gains speak for themselves: -53% reduction in MoE pre-fill latency -Up to 1.77x higher decode throughput on a 16-chip TPU v7x slice compared to a similar H200 cluster A significant milestone in efficient MoE scaling and hardware utilization!

Context

The 53% cut in Ant Ling's post about its work with the SGLang team, which builds an open-source engine for serving language models, measures one piece of the model's work rather than a whole request. The LMSYS write-up behind it, published June 17, 2026, describes Fused V2, a custom kernel written in Pallas, JAX's language for low-level TPU code, for running Ling-2.6-1T on Google's TPU v7x chips. Ling-2.6-1T is a mixture-of-experts model: each is sent to 8 of 256 specialist sub-networks plus one shared one, so 63 billion of its trillion parameters are active per token. That means scattering tokens to their experts, running each expert, and gathering the results back. The earlier kernel already fused those steps, but it streamed small slices of data through the chip's fast on-chip memory and sent partial results out to slower main memory and back, so data movement and computation overlapped little. V2 keeps tokens and intermediate results on-chip and streams each expert's weights in while the previous computation runs, hiding most of that traffic.

The kernel's prefill time, the phase that reads the prompt, fell from 5.16 to 2.42 milliseconds, but end-to-end prefill throughput improved by about 24.8%, and the per-step generation kernel's latency fell about 15%. The 1.77x figure compares 16 TPU v7x chips with 16 Nvidia H200 GPUs on SGLang's standard random-prompt with 16,384 input and 1,024 output tokens, and it applies at 512 simultaneous requests. At 128 requests the advantage is 1.29x, and the write-up says the TPUs still trail the H200s on prefill because one kernel for that phase is not yet optimized. The result is a measured gain for generation-heavy serving at high load on TPUs, not a general win over GPUs.

Terms in this piece · Glossary
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
More from Ant Ling
Recommended reads
Comments

Checking sign-in…

Loading comments…