Phobos – a tiny kernel language that now runs LLMs on a 2019 GPU
- Source
- joa-ebert.com
- Author
- joajoa
- Date
Context
Phobos is a small GPU kernel language its author built as a learning exercise, and it now runs full language-model on an older consumer card. According to the author's post, kernels are compiled at runtime and need only an NVIDIA driver, not the usual CUDA libraries; the measurements come from an RTX 2080 SUPER with 8 GB of memory running Windows.
He compares decoding speed (how fast the model writes output) with llama.cpp, a widely used open-source runner. On a 0.8B-parameter model he reports 324 tokens (small chunks of text) per second for Phobos versus 247, near parity on a 4B model, and llama.cpp about 6% faster on a GLM model. On a 35B model he reports 54 versus 30, but notes the two use different designs, since llama.cpp has no expert cache.
He cautions that the results come from one card and one operating system, that processing trails on the small models, and that llama.cpp supports far more formats. Asked whether it beats llama.cpp, he answers no.
- Inference — Using a trained model to produce an output from a supplied input.
- Open-weight model — A model whose trained weights can be downloaded and run outside the maker's hosted API.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- Prompt — Information or instructions supplied to a model; in a chat, the request you type is often called your prompt.
Checking sign-in…
Loading comments…

