AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
If you're doing RLHFReinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.Full definition → or RL fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → of language models, this explains how agents exploit reward-function flaws — modifying unit tests to pass coding tasks, sycophantically mirroring user preferences — so you can anticipate and mitigate these failure modes before deployment.
Key quotes
“Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task.”
“Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.”
“With the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge.”
