Quantization-aware distillation lifts Liquid's 4-bit checkpoints to 97% of BF16
- Source
- x.com
- Date
Today, we release updated 4-bit checkpoints for LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B trained with Quantization-Aware Distillation (QAD). These checkpoints recover accuracy lost to 4-bit quantization while retaining the low memory footprint and high decode throughput of the Q4_0 format. All four checkpoints reach roughly 97% of their BF16 averages. 🧵
Post-training quantization (PTQ) compresses an already-trained model without retraining it to adapt to quantization error, which can degrade model quality at lower precisions. QAD addresses this by simulating quantization during training and teaching the resulting quantized student to match a full-precision teacher. The final checkpoint remains a standard Q4_0 GGUF, so it requires no specialized inference runtime. (2/4)
Quality recovery is 70.6%, 73.4%, 65.5%, and 48.4% of the BF16-to-Q4_0 quality gap at 230M, 350M, 1.2B, and 2.6B respectively. The 230M and 350M QAD Q4_0 checkpoints match Q5_K_M quality within evaluation variance at a 4-33% higher decode throughput when benchmarked across a MacBook Pro, NucBox EVO-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5. The 1.2B and 2.6B QAD Q4_0 checkpoints match Q4_K_M quality at a 3-14% higher throughput on those same devices. (3/4)
The QAD GGUFs for LFM2.5-230M, LFM2.5-350M, and LFM2.5-1.2B-Instruct are available now on Hugging Face: > Run them with llama.cpp on phones, laptops, mini PCs, and single-board devices like the Raspberry Pi 5 > For more details, visit our blog: https://t.co/0894BTcOqR
Context
Liquid AI trained compressed versions of four LFM2.5 models to recover some accuracy lost when their stored numbers are reduced to four-bit precision. During training, the compressed model learns to match a higher-precision teacher while experiencing the numerical errors it will face when deployed. Method and evaluation.
The resulting checkpoints retained roughly 97% of their respective BF16 baseline scores across the company's suite. That percentage describes performance relative to the baseline, rather than the fraction of lost accuracy recovered. The files remain standard Q4_0 GGUF models, so compatible software can use the improvement without a specialized execution system.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
postLiquid AI ships LFM2.5-24B-A2B open weights for local agents
postToday we release Open d1: two open-weight multimodal models in our d1 decision…
postLiquid AI's d1 decision model adds vision at a fraction of frontier cost
articled1 is now available on @vercel AI gateway!We worked with the amazing vercel…
Checking sign-in…
Loading comments…