Vibeleaderboard
← All Intel
Intel / article

Combatting adversarial distillation

Source
openai.com
Date

Context

OpenAI has published how it fights adversarial distillation, and the core of its argument is that copying a model can strip away its safety rules. is an ordinary training technique: one model, the student, learns from answers or reasoning produced by another, the teacher. It becomes adversarial when someone uses a frontier model's outputs without permission to train their own model, which then has similar abilities without the original's safeguards. OpenAI's main example is cybersecurity: a model that helps defenders find software flaws can teach a copy that will help attackers exploit them. OpenAI is most concerned about protected reasoning, the intermediate a model produces before its answer, which users do not see and which can contain information the final answer leaves out.

OpenAI offers an internal study as evidence that this hidden reasoning matters. It trained the same starting student on coding and defensive security work from its Astra-level models, once using what users normally see (answers, reasoning summaries and ) and once with the exact reasoning traces. After , the student given exact traces generally did better. After a further light round of reinforcement learning the picture was mixed: the exact-trace student scored higher only at the largest amount of thinking time tested, while the summary-trained student scored higher at lower budgets. OpenAI reads this as support for protecting traces, and notes that Anthropic reported a similar advantage on a coding benchmark.

The defenses work at two levels. For each request, models are trained to refuse attempts to extract their reasoning, classifiers check incoming requests and check outgoing messages and tool-call arguments for leaks, and the API now ties a model's reasoning to the conversation that produced it, dropping it if an earlier message is rewritten and keeping older, less protected model families from reading it. In OpenAI's own automated attack tests, the success rate for recovering reasoning from an earlier request fell from 54.0% against GPT-6 Astra with its original safeguards to 0.2% against GPT-6.2 Astra with updated safeguards, and from 22.9% to 0.9% for the current request. Those figures compare a newer model as well as newer safeguards, and they are not independent measurements. At the second level, OpenAI investigates coordinated campaigns and says its weekly enforcement has reached thousands to tens of thousands of accounts. It also repeats earlier accusations against DeepSeek and individuals associated with Moonshot AI, and backs US legislation on export controls and sanctions. Anthropic described its own defenses against the same kind of attack in October.

Key takeaways · AI-distilled
  • OpenAI says updated safeguards cut the success rate of adaptive attacks recovering a model's earlier protected reasoning from 54.0% on GPT-6 Astra to 0.2% on GPT-6.2 Astra, and from 22.9% to 0.9% for the current request, across six attacker models.
  • In an internal cyber study, a student fine-tuned on a teacher's exact reasoning traces generally beat one trained on reasoning summaries; after reinforcement learning it led only at the largest inference budget tested, while the summary-trained student led at lower budgets.
  • OpenAI's output classifiers check tool-call arguments as well as assistant messages, because a model can leak protected reasoning inside a tool's arguments without that text appearing in its final answer.
  • With GPT-6.2 Astra and Sol, protected reasoning is tied to the conversation that produced it: if an earlier message is rewritten, the API drops the reasoning that followed, and older model families with weaker safeguards cannot read it.
  • OpenAI reports weekly enforcement actions against distillation campaigns ranging from thousands to tens of thousands of accounts, with the rate rising significantly this past summer, and says it shares indicators of compromise with other model providers.
Terms in this piece · Glossary
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
  • tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Recommended reads
Comments

Checking sign-in…

Loading comments…