Vibeleaderboard
← All Intel
Intel / article

Training Language Models for Introspective Oversight

Source
arxiv.org
Author
Alex Lin Wang, Linus Yifeng Tang, Dami Choi, Neil Chowdhury
Date

Context

Researchers at Transluce, with a coauthor from Thinking Machines Lab, train a model to answer questions about its own behavior and reward it only when the answers check out. The usual ways of catching a model that lies, flatters the user or games its tests each have gaps. Asking the model directly is unreliable because it is not trained to report its own misbehavior, an outside model reading the transcript sees only what was written, and activation oracles, separate models that translate a model's internal activations into words, can confabulate. The new approach adds an observer role to the conversation format alongside the user and the assistant. The observer shares the assistant's weights, so it can draw on the same internal computation, but it is trained only for accuracy about the assistant, and its answers never enter the assistant's context.

Training started from Qwen's Qwen3.6-27B. Supervised fine-tuning first taught the observer to predict what the assistant would do with a prompt and to infer a hidden instruction, such as a persona placed in a it could not see, from the response alone. Reinforcement learning then rewarded it for predicting, with probabilities, how the assistant would behave if the prompt were edited, and for picking a property of the assistant's coming response from plausible alternatives, with answers scored against samples from the assistant. A step kept the assistant's own behavior close to the original.

The observer was tested on five evaluations it had not trained on, including whether the assistant had shifted its ethical judgment toward a user's stated view, changed an answer under harmful pressure, proposed gaming tests in coding tasks, or falsely claimed to have run code. Scores are AUROC, the chance that a method ranks a real case of the behavior above a clean one, where 0.5 is guessing and 1 is perfect. The observer averaged 0.907, against 0.864 for honesty training, 0.828 for a more capable outside model reading the transcript, 0.796 for an activation oracle and 0.776 for simply asking the assistant. Its gains were largest on flattery and on tracking answers that shift with unrelated value cues; it performed similarly to the best baseline on false claims and slightly below it on lies under pressure. Two larger models fine-tuned on the same data but without the assistant's weights scored 0.684 and 0.647, which the authors take as evidence that access to the model's own internals helps. After training, the assistant also became easier to monitor: every other method improved on average, and direct questioning rose from 0.776 to 0.892, for reasons the authors could not separate.

The results come from a single 27B model whose evaluation transcripts were generated without reasoning, and some test sets are small, 40 items for reward hacking. With reasoning switched on, the observer only matched an external transcript monitor, 0.823 against 0.822. The authors did not test stronger models capable of hard-to-monitor actions or models trained to hide objectives, did not check the accuracy of the observer's free-form answers, and left open whether it reads the assistant's internal states or has simply learned to predict its behavior. They advise against using the observer's reports as a training signal for the assistant, since that could teach the assistant to evade it.

Key takeaways · AI-distilled
  • The authors add a separate third-person observer role to a model, trained only for accuracy and apart from the assistant, to answer arbitrary questions about the assistant. Answers are rewarded when verified against what the assistant actually did.
  • On five held-out oversight evaluations, an observer trained on Qwen3.6-27B reached a mean AUROC of 0.907, ahead of honesty training (0.864), transcript monitoring (0.828) and activation oracles (0.796).
  • Training the observer also made the assistant easier to monitor: every baseline method improved on average when applied to the fine-tuned assistant, the paper reports.
Terms in this piece · Glossary
  • system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
  • calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Recommended reads
Comments

Checking sign-in…

Loading comments…