Intel / article

Understanding Likelihood Over Optimisation In Direct Alignment Algorithms 2024 10 18

Source: cohere.com
Author: Cohere
Date: Published Jun 6, 2025

Context

Making a model more likely to repeat preferred training answers can eventually reduce its performance. The authors of this alignment study examined direct methods, which tune a toward preferred completions and away from rejected ones. They report that a higher likelihood of preferred completions, or a larger margin over rejected ones, did not necessarily improve results and could degrade them. Their analysis finds that higher likelihood correlates with better memorization of factual patterns, while somewhat lower likelihood tends to improve output diversity and generalization to unseen scenarios. For the methods studied, success on the training objective was therefore an imperfect guide to how well the model would handle new cases.

Terms in this piece · Glossary

alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.

Read the source cohere.com