Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

This paper asks what actually breaks when a mixture-of-experts model is made cheaper to serve by giving prompt tokens less compute, and its answer is that most of the damage lands on a handful of tokens near the ends of the prompt (arXiv). A mixture-of-experts (MoE) model contains many expert sub-networks in each layer, and a router sends each token through only a few of them, its top K. Prefill is the step where the model reads the whole prompt before writing its answer. The cost-cutting trick studied here routes each prompt token through half its usual experts during prefill, for example 4 instead of 8 in Qwen3-30B-A3B, while generated tokens still use the full K (paper HTML).

Across four MoE families tested on short-answer question answering, scored with F1, which rewards answers that match the reference and drops when a correct answer arrives wrapped in extra words, the author finds that for three families the loss at half the experts is mostly about answer form. The model still finds the answer but stops answering briefly. In Qwen's Qwen3-30B-A3B the loss is made at the two chat-template tokens just before the answer, and in DeepSeek's DeepSeek-V2-Lite at the first 8 to 32 prompt tokens, where its instruction sits. Measured by answer likelihood, those positions account for about half and about a fifth of the damage. Restoring other suspected culprits, the so-called super experts or massive activations, recovers at most about a quarter of the loss, well below what keeping the sensitive positions at full experts achieves.

On real prompts, keeping about 4% of tokens at full experts on the task interface kept F1 within one point of the unmodified model, while placing the same number at random in the documents was about 3 points worse. That conclusion rests on two of five prespecified primary comparisons; the others were undecided or not run. Because the sensitive tokens sat at the prompt ends in every layout the author found, the paper cannot tell whether a serving rule should simply protect the ends of a prompt or look for application markers like chat templates. The speed gain is also modest and depends on length: time to first token falls about 14% on 11K-token prompts and not at 2K tokens.