Colibri
github.com/justvugg/colibri- Category
- Developer Tools
- Rank
- No. 707Tools index
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- GitHub
- 44.2k stars
- Latest release
- v2.0.0
- Date
About
Colibri is a pure-C inference engine that runs large mixture-of-experts language models, from 744B up to 2.8T parameters, on consumer hardware by treating disk, RAM, and VRAM as a single memory hierarchy and streaming experts from storage. It supports eight model families, including GLM-5.2/5.3, Kimi K3, and DeepSeek V4 Flash, with zero external engine dependencies.
What it does
A small C program that answers prompts with very large mixture-of-experts models on an ordinary desktop. Such a model only uses a sliver of its weights for each token, so Colibri keeps the always-used part in memory (about 10 GB for the 744B GLM model) and reads the rest, roughly 370 GB, off an SSD as each token asks for it. A cache learns which expert blocks you use most and keeps them in RAM or GPU memory. You talk to it through a terminal chat, a browser dashboard, or a local server that speaks the OpenAI and Anthropic API formats.
Why it's ranked here
Few projects let a single workstation run a 744B-parameter model at all, and this one publishes its own limits. The benchmark page opens with "This is not fast": the author's own box managed 0.05 to 0.1 tokens per second cold, and community rows range from 0.07 on a 24 GB WSL2 machine to about 2 tokens per second on 128 GB Macs with the Metal backend. It is Apache 2.0, marked beta, ships prebuilt downloads for Linux, macOS and Windows, and states a rule that placement may cost speed but never changes precision. A serious research engine, slow by design, honest about it.
What's good
The documentation measures instead of promising. Hardware tables name the disk, RAM, settings and expert hit rate behind each number, and estimates are labelled as estimates. The GPU notes warn that a device line in the log does not prove any work ran on the GPU, and tell you which counter to check. The API server refuses images, log probabilities and token penalties with an explicit error rather than ignoring them. A diagnostic command reports missing compilers, model files or libraries. The default build needs no GPU and no BLAS library, and the quick start assumes no prior experience with C or CUDA.
Tradeoffs
Speed is set by your SSD. The quick start says a slow or shared disk can fall well under one token per second, and the model needs roughly 380 GB free. The server handles one generation at a time and queues the rest (eight waiting by default). GPU results round differently from the CPU path, so output is not token-identical across backends, and the Windows AMD build was validated on one machine with no hosted CI. There is no prebuilt ARM64 Linux download. The seed field is accepted but has no effect, and the older per-row int4 model downloads are documented as causing looping, never-ending replies.
How to use it well
Best for someone with a fast NVMe drive and 24 GB or more of RAM who wants a frontier-scale open model fully offline, or who wants to study how expert routing behaves. Grab the recommended group-scaled container from the quick start rather than an older per-row one. Run the diagnostic and placement-plan commands first, measure your disk with the bundled benchmark, then record usage and pin the hottest experts into spare RAM. Lowering top-p to 0.85 reads fewer bytes per token. Set an API key before exposing the server. It does not suit concurrent multi-user serving or latency-sensitive interactive work.
Technical notes+
The pip package colibri-engine (pyproject.toml, requires-python >=3.10, Development Status 4 - Beta) installs a coli console script whose entry, colibri/cli.py, just locates the c/ directory next to the package and runs the launcher script there via runpy, exiting with a clone-and-install hint if it is missing. Optional extras split conversion (numpy, huggingface_hub), an oracle (torch, transformers, safetensors) and benchmarking. GPU_BACKENDS.md describes one CUDA source compiled for NVIDIA or AMD through a shim header mapping 14 runtime symbols; CUDA=1 and HIP=1 are opt-in and Linux-only. On Windows, c/backend_loader.c loads the backend DLL with LoadLibrary and GetProcAddress, returns a not-initialized sentinel so the engine falls back to CPU when it is absent, and in HIP mode binds amdhip64_7.dll only from COLI_HIP_RUNTIME_DIR. docs/FORMATS.md registers weight format ordinals 0 to 8, inferred from byte arithmetic rather than stored in the container. docs/api.md documents the bounded FIFO admission queue and Prometheus metrics under the colibri_scheduler_ prefix.
Observed
- License
- Apache License 2.0
- Language
- C inference engine; Python launcher, converter and HTTP gateway
- Packaging
- pip package colibri-engine with a coli console script; prebuilt archives for Linux x86_64, macOS and Windows; source build via make
- Interfaces
- CLI (chat, serve, web, doctor, plan, convert, bench); OpenAI-compatible HTTP API; Anthropic Messages endpoint; Prometheus metrics
- Platforms
- Linux, macOS, Windows; ARM64 Linux by source build only
- GPU backends
- Optional CUDA, HIP/ROCm, Metal and Vulkan; default build is CPU-only
- Declared maturity
- Development Status 4 - Beta
- Stated minimum hardware
- About 16 GB RAM and about 380 GB free disk for the int4 GLM-5.2 model
- Windows HIP validation
- One hardware configuration; no hosted CI coverage
Read from README.md, pyproject.toml, LICENSE, colibri/cli.py, docs/quickstart.md, docs/FORMATS.md, GPU_BACKENDS.md, docs/api.md, docs/benchmarks.md, c/backend_loader.c.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.