Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters

Qwen's Qwen1.5-MoE uses only part of its learned parameters for each input, but the full model is much larger than its 2.7B active-parameter label suggests. The architecture and size table list 14.3 billion total parameters, with 2.7 billion active.

The model divides computation among expert components. Four shared experts are always active, alongside four selected from 60 other experts. This reduces computation at each step while retaining a larger collection of weights. The team reports competitive results against contemporary 7B models and explicitly acknowledges higher memory requirements. Active parameter count therefore helps explain compute use, but cannot stand in for the model's storage footprint.