DistillKit ships with logit compression and pre-captured teacher data

Training a smaller 'student' model to imitate a larger 'teacher' works best when the student sees the teacher's full probability distribution over possible next tokens (its 'logits') for every token, not just the teacher's single chosen word. But storing that full distribution across a large training set is expensive, Arcee says prohibitively so, since it means saving a large set of numbers for every token in the dataset.

Arcee's open-source DistillKit toolkit compresses those logits using polynomial approximation, error-diffusion quantization, and bit-packing, to roughly 300 bytes per token, which Arcee says is about 0.15% of the uncompressed size with minimal reported quality loss. Alongside the toolkit, Arcee is releasing pre-captured logits from three teacher models, Qwen3-235B, DeepSeek V3/R1, and DeepSeek V3 Base, so a team can start an offline distillation run without first generating and storing that teacher data itself.

Arcee says DistillKit is the same toolkit it used internally to train its earlier Virtuoso, SuperNova Medius, and Blitz models, and that an entire distillation run, teacher, student, loss functions, compression, and training arguments, can be configured from a single YAML file.