Misalignment Has a Personality: A Big Five Account of Emergent Misalignment
Source
arxiv.org
Author
Hasibur Rahman, Smit Desai
Date
Why it matters
Corpora that induce broad misalignment share a measurable Big Five signature — lower agreeableness and conscientiousness, higher extraversion and neuroticism — detectable in the training data before fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → imprints it on the model.
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
articleOptimismBench: Forecasting Bias and the Alignment Effect in Language Model JudgmentSeonglae Cho, Adriano Koshiyama
articleAlignment Forecasting: Predicting Misalignment From Training DataChen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky, Tomek Korbak
articleConstitutional Midtraining: Content Presence Drives Alignment GainsDesiree Cho, Cameron Tice, Bernie Hogan, Hunar Batra, Puria Radmard, Jun Zhao, Nigel Shadbolt