How Goodfire used Ai2’s open post-training stack to trace unwanted model behavior

Goodfire, an AI interpretability company, used Ai2's open training materials to investigate why a model became more willing to answer harmful requests. Preference training teaches a model from pairs of preferred and rejected responses. Ai2 reports that some pairs rewarded compliance with harmful requests while rejecting refusal, and Goodfire traced part of the behavioral regression to those examples. Research account.

Access to the examples, intermediate model checkpoints and reproducible training recipes let researchers intervene in the process and test targeted changes. The contribution is a way to investigate which training choices changed behavior, giving developers more to work with than a final model that fails a safety test.