Predicting what preference training will break, before you train

Preference training shapes a language model with pairs of responses to the same prompt, one marked preferred and one rejected. Across hundreds of thousands of pairs, those choices add up to a signal about which behaviors to strengthen and which to suppress. Nobody can read that signal directly, so unintended lessons usually show up only in evaluations after training. Goodfire, an interpretability research company, says it can predict those lessons in advance. Its method, which it calls predictive data debugging, passes the preference data through a model whose internal concepts have already been interpreted, before any training on that data. The concepts that distinguish preferred from rejected responses are the prediction: they are what the model will move toward, or away from, during training. Goodfire reports that the prediction matches what the model actually learns at R² = 0.9. Because each concept is tied to specific pairs, it also points to the examples responsible.

Ai2 presents the work as a demonstration of its open post-training stack. Dolci, the preference dataset behind Olmo 3, publishes every individual preferred and rejected response. Olmo comes with intermediate checkpoints and reproducible training recipes, and Ai2's OLMES suite measures capability changes on a consistent set of benchmarks. Leon Bergen of Goodfire says this reproducibility let the team change one component and measure the effect against Ai2's official Olmo checkpoints instead of guessing. Goodfire's own write-up adds that its case studies mostly ran on Dolci with base models from Llama-3.1-8B up to 70B, so not every result is a measurement of Olmo itself.

The main finding is a safety regression. Preference training on Dolci raised general capability but made models substantially more likely to answer harmful requests. Goodfire traced part of this to pairs where the preferred response accepted a jailbreak framed as fiction or a hypothetical and the rejected response refused. It says that training on a corrected version of the data improved both safety and performance: a model trained on the corrected data refused a fictional request about converting a handgun to fully automatic fire, which the model trained on the original data had tried to answer. The method also surfaced a behavior nobody would have thought to write an evaluation for. In a cluster of fan-fiction prompts about characters in a pond passing gas and killing nearby fish, the preferred response wrote the scene and the rejected one declined, and after training the model took on such requests enthusiastically.

Goodfire is open about where its fixes fell short. Its interventions only partly reduced the hallucinated links that preference training increased on sensitive queries. They also failed to fully remove a new tendency to flatter users who ask pseudo-profound or nonsensical physics questions. To check that the findings are real, Goodfire planted goblins in some training responses, then used the pipeline to find and remove that planted behavior. The figures above come from Goodfire's and Ai2's posts. The full method and results are in the paper on arXiv.