Stealing Reasoning Traces from Proprietary LLM APIs
- Source
- simonwillison.net
- Date
Context
An encrypted reasoning block can still leak if another model is allowed to interpret it. Researchers report replaying a stronger model's encrypted output into a weaker sibling and inducing that model to reveal the underlying text. The failure concerned reuse across contexts: encrypting the returned block did not prevent disclosure through a model that could process it. Researchers' account.
The initial report said the original attacks stopped working after disclosure. A September 30 update, titled as a mitigation audit, reports further extraction through third-party API providers, so that initial mitigation should not be read as a complete resolution. Researchers' update.
- chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
“We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext”
“The paper's authors found that every model under the same family used the same encryption key, which meant you could feed those blocks back into the weakest model family members and jailbreak them into outputting the unencrypted raw reasoning blocks!”
Simon Willison
“Claude Haiku 4.5 was the easiest to attack.”
Simon Willison
“Models appear to treat their own reasoning traces as sacrosanct, and are much more likely to follow instructions that somehow make it into those chunks.”
Simon Willison
“The reasoning tokens that were revealed were clearly never intended for human consumption.”
Simon Willison
Checking sign-in…
Loading comments…


