llama.cpp v0.9.0

llama.cpp, the open-source engine for running language models on local hardware, shipped v0.9.0 with a change aimed at serving several conversations at the same time. A model keeps the keys and values it has computed for earlier tokens in a KV cache so it does not recompute them for each new token. When the llama.cpp server handles several sequences at once, it can hold them all in one shared cache, called the unified KV cache, or give each sequence its own. In the pull request behind the new flash_attn_ext_rows operation, its author says the unified cache "pays a big penalty" because the attention mask has to filter out every other sequence's tokens. The new operation instead hands each sequence's queries an index list of the cache rows that belong to it, so attention works only on those rows.

The author's benchmark ran four parallel sequences with prompts of about 100,000 tokens each on an RTX Pro 6000. Prompt processing for Gemma-26B-A4B at Q8_0 rose from about 2,982 to about 8,626 tokens per second, and for Qwen3.8-27B at Q8_0 from about 1,106 to about 2,778. Token generation moved less, from about 273 to 288 and from about 90 to 102 tokens per second. The author describes the result as roughly on par with giving each sequence its own cache, which matches the release notes' claim that the change restores the prefill speed of the non-unified cache. These figures come from one configuration on one GPU. During review, Georgi Gerganov said Metal, the backend for Apple hardware, gains little or nothing, because its existing kernels already skip masked-out blocks quickly.

The second headline item adds the OCP FP8 E4M3 format to ggml, the tensor library llama.cpp is built on. FP8 stores each number in 8 bits; E4M3 is the variant with 4 exponent bits and 3 mantissa bits. The FP8 pull request gives its reason as E4M3 being the variant most commonly used for inference, with E5M2 more common in training. The release adds a matching model file type, and the PR includes conversion fixes for checkpoints that already ship FP8 weights, though its description leaves fuller handling of scale tensors in ggml to later work. Only the saturating mode is implemented; the PR says the non-saturating mode, which preserves NaN values, is not. The PR includes a fast CPU kernel, leaves CUDA and Vulkan support to separate pull requests, and reports no quality or speed measurements, so the format's effect on output quality relative to existing quantizations is not established by these sources.

The rest of the release is support work. It adds four architectures: the Clover-35B-A3B-Preview diffusion model in autoregressive mode, Limite 1B, Metis-1.6 and Mistral Large 4. Audio support grows with Chatterbox text-to-speech, Nemotron 3.5 Transcribe for offline transcription in 40 languages with speaker attribution, and predefined Pocket TTS voices that do not need voice cloning. Gemma 4 E2B and E4B get graph optimizations that the changelog calls massive prefill gains without giving figures. The bundled ggml v0.29.0 also adds a batched tensor copy API, implemented for Metal and CUDA, and initial multi-node tensor parallelism using NVIDIA's NCCL communication library.