Training an LLM by showing it answers people prefer – and ones they don’t – can…

• Ai2 frames the problem as teams discovering unwanted behavior changes only after preference training finishes, then working backward from eval scores to guess which of hundreds of thousands of examples caused them.
• Goodfire's "predictive data debugging" estimates which behaviors a preference run will strengthen or suppress before committing compute, built on Dolci preference data, Olmo intermediate checkpoints and reproducible recipes, and OLMES capability evals.
• In one experiment the regression was higher compliance with harmful requests framed as fiction or hypotheticals. Goodfire traced part of it, not all, to specific Dolci preference pairs.

• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

You can trace a behavior regression from preference tuning back to specific training pairs, and estimate it before spending compute on a full run.

• postGibberish wins: a game exposes the limits of prosocial-behavior scoring
• postAi2 uses psychometrics to audit what LLM safety benchmarks measure
• postMolmoMotion forecasts object motion in 3D from a few frames and an instruction
• postA substitution test shows LLMs answer drug questions from name patterns

• articleHow Goodfire used Ai2’s open post-training stack to trace unwanted model behaviorallenai.org
• postTracing Olmo's reasoning back to the text that produced itAi2
• articleHow a Georgia Tech team used the open Olmo stack to trace social reasoningallenai.org

Checking sign-in…

Loading comments…