llama.cpp Multi-GPU MoE Fork
github.com/neurall/llama.cpp- Category
- Developer Tools
- Rank
- No. 3025Tools index
Previous survey · No. 1554 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- GitHub
- 16 stars
- Latest release
- release-b14929-8e4c2d4ff-ram-pool
- Date
About
A fork of llama.cpp that improves multi-GPU scaling for mixture-of-experts models, delivering roughly 2 to 4 times faster inference on models too large to fit entirely in VRAM. It targets users running big MoE models split across multiple GPUs where the base project's scaling falls short.
At a glance
This llama.cpp fork explores running mixture-of-experts models larger than GPU memory by caching frequently used experts in VRAM and computing misses on the CPU. It is worth benchmarking if host RAM is available and your current CPU-heavy setup is slow.
Good fit
Local inference experimenters who can compare against upstream using the same model and prompts.
Consider something else when
Use a supported build if you need predictable compatibility across devices and model families.
Start here
Use an approved model you already have. Compare prompt processing and generation separately against upstream while recording RAM, VRAM and output correctness.
Cost and requirements
A source build for your hardware, an appropriate model file and enough system RAM to hold its experts.
Limits to know
The author distinguishes expert caching gains from multi-GPU gains. Prefill support and speedups depend on the model, prompt length and hardware.
Sources checked 2026-11-05. Source-based assessment.
Why it made the leaderboard
Speeds up local inference for MoE models that don't fit in a single GPU's VRAM, letting practitioners run larger open-weight models faster across existing multi-GPU rigs.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.