Virtuoso Lite Virtuoso Medium V2 Distilling Deepseek V3 Into 10b 32b Small Language Models Slms
Source
arcee.ai
Author
Arcee AI
Date
Context
Arcee AI's Virtuoso-Lite and Virtuoso-Medium-v2 transfer information from DeepSeek-V3 into smaller models with 10 billion and 32 billion parameters. Arcee says its training uses the larger model's raw output scores, giving the smaller models more information to learn from than finished answers alone. Release and training account.
For Virtuoso-Lite, the process included temporarily adapting how text is divided into model units to match the teacher, then reverting that text interface and applying further alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → training. This is a concrete example of transferring a larger model's behavior across different model designs. Both releases were offered under Apache 2.0. Distillation details.
Terms in this piece · Glossary
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
