Alignment Forecasting: Predicting Misalignment From Training Data
Source
arxiv.org
Author
Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky, Tomek Korbak
Date
Context
Alignment Forecasting asks whether training data can warn of behavioral problems before a model is trained on it. The proposed forecaster rates how strongly examples encourage misconduct, then combines that assessment with the target model's previous tendencies and the frequency of the particular failure.
The authors report predictions better than chance and use the resulting signals to remove problematic training examples. Models trained on the filtered data usually improved on their multiple-choice alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, but benefits in open-ended conversations remained unclear. The result makes advance screening a research direction while leaving its usefulness for realistic conversations unresolved.
Key takeaways · AI-distilled
AlignmentForecastBench has over 5,000 questions spanning 17 target models, 32 fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → datasets and 16 failure modes. A forecaster outputs the probability that fine-tuning on a dataset would meaningfully increase a given failure mode.
Frontier models prompted directly forecast poorly. The proposed scaffold has an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → rate how strongly and broadly a dataset pushes toward misbehavior, then a simple learned model combines that rating with the failure mode's base rate and the target model's prior tendency.
The scaffold forecasts well above chance and beats both a model fine-tuned on the task and a forecaster allowed to see how weaker models behaved after training on the same data, the authors report.
Removing examples the scaffold flags from real post-training data such as UltraChat gave more aligned models on a multiple-choice evaluation in most cases, but the benefit in open-ended conversation is unclear and the results cover only SFT.
Terms in this piece · Glossary
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
articleOptimismBench: Forecasting Bias and the Alignment Effect in Language Model JudgmentSeonglae Cho, Adriano Koshiyama
articleOpenAI-HuggingFace: A Reproduction & Lessons for Alignment TestingStewart Slocum, Malayandi Palan, Christopher Chute, Michael Kim, Benjamin Van Roy
videoAlignment faking in large language modelsAnthropic