Today, we release updated 4-bit checkpoints for LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B trained with Quantization-Aware Distillation (QAD). These checkpoints recover accuracy lost to 4-bit quantization while retaining the low memory footprint and high decode throughput of the Q4_0 format.
All four checkpoints reach roughly 97% of their BF16 averages.
🧵
Post-training quantization (PTQ) compresses an already-trained model without retraining it to adapt to quantization error, which can degrade model quality at lower precisions. QAD addresses this by simulating quantization during training and teaching the resulting quantized student to match a full-precision teacher.
The final checkpoint remains a standard Q4_0 GGUF, so it requires no specialized inference runtime.
(2/4)
Quality recovery is 70.6%, 73.4%, 65.5%, and 48.4% of the BF16-to-Q4_0 quality gap at 230M, 350M, 1.2B, and 2.6B respectively.
The 230M and 350M QAD Q4_0 checkpoints match Q5_K_M quality within evaluation variance at a 4-33% higher decode throughput when benchmarked across a MacBook Pro, NucBox EVO-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5. The 1.2B and 2.6B QAD Q4_0 checkpoints match Q4_K_M quality at a 3-14% higher throughput on those same devices.
(3/4)
The QAD GGUFs for LFM2.5-230M, LFM2.5-350M, and LFM2.5-1.2B-Instruct are available now on Hugging Face:
> Run them with llama.cpp on phones, laptops, mini PCs, and single-board devices like the Raspberry Pi 5
> For more details, visit our blog: https://t.co/0894BTcOqR
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Why it matters
The files stay plain Q4_0 GGUF, so llama.cpp runs them unchanged while quality sits near larger quant formats and decode runs faster on laptops, phones and a Raspberry Pi 5.
