distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → means training a weaker model on a stronger model's outputs. In post-training it takes two forms: a general data engine for instructions, preference data and RL verification, and a targeted transfer of one agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → such as math or coding into a smaller model.
The misconduct in the Chinese-lab cases was not distillation but jailbreaking, hacking and identity-spoofing APIs to extract data the provider never meant to expose, such as reasoning traces. Lambert argues those actions should be named abuse, since distillation is standard.
Attribution gets muddy fast. You use a GPT API to bootstrap data for a small specialist model like olmOCR, that model generates a huge corpus, then you train a third model from scratch on it. Whether the final model is "distilled from GPT" has no clean answer.
Closed-model terms of service generally forbid using the API to build a competing model, but the clause has gone almost entirely unenforced. Before the recent Chinese cases, only one prominent instance of restricted corporate accounts existed.
Kevin Xu's counterintuitive case: if Chinese labs stay dependent on distillation as their shortcut to the frontier, they never build the techniques needed to lead. Cutting off the crutch buys a short-term US lead and may hand them a stronger long-term trajectory.
Terms in this piece · Glossary
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
Why it matters
If you build on or distill from open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → models, this clarifies how 'distillation attacks' rhetoric could translate into regulation that restricts a workflow you likely depend on. It cuts through conflated terminology so you can evaluate the actual policy risk to the open-weight ecosystem.
Key quotes
“What these few labs are doing should be referred to as jailbreaking or abuse, rather than distillation.”
“Associating all of distillation with these attacks, which is to date an industry standard for post-training, from open and closed models alike will be a massive own goal.”
