DistillKit is officially out.
The same toolkit we used to train Virtuoso, SuperNova Medius, and Blitz. Online distillation, offline distillation, advanced compression. All of it.
Storing teacher logits for offline distillation is expensive. Like, prohibitively expensive.
Our compression system: polynomial approximation + error-diffusion quantization + bit-packing.
~300 bytes/token. 0.15% of uncompressed size, with minimal quality loss.
Configure your entire distillation run in YAML.
Teacher, student, losses, compression, training args—all in one file.
distillkit config.yaml
That's it.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Offline distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → usually stalls on the cost of storing teacher logits. Compressing them to about 300 bytes per tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →, plus ready-made teacher data from Qwen3-235B and DeepSeek V3/R1, makes training your own student model practical.
Read the full thread on X ↗
