Vibeleaderboard
← All Intel
Intel / article

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

Source
huggingface.co
Date

Context

Tarek Ziadé has run a roughly 180-billion-parameter model on a laptop with 32 GB of memory through the standard Transformers library, at about 14 tokens per second and within about 6 percent of Strata, a specialised engine built for the job. In his write-up on the Hugging Face blog, he shows how this works for (MoE) models. Each layer of an MoE model contains many expert sub-networks, and a router picks only a few of them for each token. Since experts make up most of the weights, they can stay on the SSD and be read in only when the router chooses them.

The target was Qwen3.8-Flash-Next from Qwen, which has 48 layers of 512 experts with 10 active per token, 121 GB of experts at 8-bit precision, and a 51 GB n-gram table on top (Qwen's launch announcement describes it as 125 billion parameters plus 51 billion of n-gram embeddings, with 6 billion active per token). Transformers could not handle that before. Its existing offloading copies a layer's entire expert bank to the GPU on every token in order to use 8 or 10 of them. In Ziadé's comparison on a smaller MoE model and a 24 GB data-centre GPU, that path decoded at 0.16 tokens per second, against 82 to 88 for Strata and 63 to 64 for llama.cpp. On a Mac, where the CPU and GPU share memory, offloading to the CPU frees nothing anyway.

The central fix is a slot pool: one preallocated block of GPU memory that holds the cached experts for every layer, plus a per-layer table mapping each expert's ID to its slot. Translating the router's choices into slot numbers makes the cache look like an ordinary expert bank, so the model's existing fast kernel, which runs all of a layer's chosen experts in one call, works unchanged. On a small test model, that lifted a naive one-expert-at-a-time loop from 10.1 to 34.6 tokens per second, against 38.1 for the unmodified model. For the large model, experts were compressed to 3 bits (49 GB on disk, 12 GiB cached), the n-gram table is read from disk row by row, and the next two layers' routers run early to guess which experts to prefetch. That first version reached about 10 tokens per second. Most of the later gains came from outside the experts, including forcing PyTorch to start queued GPU work sooner, custom Metal kernels and Strata's own weight files, reaching 13.9 tokens per second against Strata's 14.8, with slightly faster prompt processing.

The result is a capability demonstration on one machine, not a finished feature. Ziadé says about 14 tokens per second is still too slow for his coding- work. Converting the model takes about five hours, and a 6,000-token prompt peaks at 25.4 GiB, so he has to close his browser to avoid swapping. The remaining gap is mostly about 10 milliseconds per token of the GPU waiting on Python, and PyTorch does not expose the control he would need to remove it. The generic code sits in a Transformers pull request whose API is still under discussion, and he notes that AI wrote the code while he chose what to try. That pull request covers MoE models with the standard expert layout; the n-gram table handling and Metal kernels this model needed live in a separate package on top of it.

Terms in this piece · Glossary
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
Read the source huggingface.co
Recommended reads
Comments

Checking sign-in…

Loading comments…