The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]
Source
youtube.com
Author
Latent Space
Date
Context
Reinforcement fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → can use the correctness of an answer as its learning signal. This NeurIPS 2024 talk explains the approach through OpenAI's then-emerging reinforcement fine-tuning offering: the model produces answers, a grader scores them, and training reinforces behavior that earns better scores.
The speaker gives a concrete example of a checkable requirement: every sentence must begin with a specified letter, which a program can verify. The key design question is how to judge success reliably. More complicated tasks need graders that handle their answer formats and requirements, making evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → part of the training mechanism.
Key takeaways · AI-distilled
Lambert, drawing on his interview with Meta's Ross Taylor, argues chain of thought is sensible because a model computes each tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → in one forward pass with no explicit place to hold intermediate values, so written-out steps act as that storage.
Lambert says to take OpenAI at face value that o1 is very large-scale RL on verifiable outcomes. In his view, theories involving process reward models, self-play or Monte Carlo tree search rest on prior literature, not on evidence OpenAI has shown.
He finds the community o1 relatives from DeepSeek and Qwen much narrower than o1: the DeepSeek model he tried sometimes declined a code question because it was meant only for math or code, while o1 handles far more tasks.
Per Lambert, RL reuses data heavily: it samples several completions per prompt (say eight) to grade or contrast, and its flexible loss allows many passes over the same prompts, which is why reinforcement fine-tuning can work from dozens of labeled examples.
Reinforcement fine-tuning data is a prompt plus a checkable answer, not a completion or chosen/rejected pair. Lambert says gentle RL on such data adds narrow skills without degrading the base model, though binary rewards fit code quality poorly.
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
