Clip / AI Tools

Algorithmic sycophancy: RLHF amplifies the composite

From The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen · ≈19:49

Explains why the obvious fix fails — human raters were shaped by the same dominant narrative, so preference optimization rewards handing users the version they already believe in.

What’s in it

Explains why the obvious fix fails — human raters were shaped by the same dominant narrative, so preference optimization rewards handing users the version they already believe in.

Watch this moment

Clip transcript

It amplifies it. And the reason is structural. Human raters evaluate outputs using their own conceptual frameworks. And their frameworks were built by the same culturally dominant narratives that saturate the corpus. The rater grew up with the same Hamilton that you did. So, when alignment optimizes for human preference, it optimizes for outputs that conform to the rater's already mythologized experience. This is a documented failure mode. We call it algorithmic sycophancy. And here, it has a specific target. The model is rewarded for having handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post training. Post training reinforces it. And every sufficiently salient historical figure that gets rendered by default as a cultural composite.