Vibeleaderboard
← All Intel
Intel / article

Lambert maps open problems in post-training and RL compute

Source
natolambert.com
Author
Nathan Lambert
Date

Context

Nathan Lambert, executive director of the research lab Trillium Labs, uses a talk he gave at Cornell on November 20 to argue that frontier post-training is slipping out of reach for open researchers, and to set out six problems they can still work on. Post-training is the stage that turns a base model, which only predicts the next word, into a useful assistant, through finetuning on examples, learning from human preferences, and reinforcement learning (RL) against answers that can be checked automatically. In the post sharing the slides, Lambert calls the list largely a statement of Trillium's goals.

His evidence for the cost problem is a table comparing the RL compute spent on a model with its compute, in GPU-hours. For DeepSeek R1, released in January 2025, RL took about 0.14 million H800 GPU-hours against 2.66 million for pretraining, around 5 percent; DeepSeek's V3 technical report gives the pretraining figure as 2.664 million. For a model the deck calls Beam, from October 2026, RL used about 7.1 million GB300 GPU-hours, more than 1.7 times its pretraining, and for Thinking Machines' Inkling 2 Small the deck estimates about four times, noting that Thinking Machines reported the pretraining duration but not the GPU count. Lambert adds that GPU-hours overstate RL's share of the actual computation, because RL spends much of its time generating samples. He puts doing post-training well at on the order of $50 million or more a year and compares the situation to pretraining at the start of 2023, a problem he calls solvable with enough funding and commitment.

He orders the six problems from least to most compute-intensive, which is what makes the list usable for groups without frontier budgets: harness specialization and design, meaning the software around a model that manages its and tools; character training, how a model gets its personality and values; open-source RL environments; reasoning efficiency in RL; scaling laws for RL; and scaling laws. Several come with his own recent work as an example. In Never Give Up, he and co-authors report that standard RL training improves results on problems a model already solves easily far more than on hard ones, and propose to keep sampling a hard problem until one attempt is correct, which they say moves compute toward the harder problems (paper). Lambert says making those RL comparisons fair was "shockingly hard." For RL environments, he argues that post-training lacks the shared open data that seeded pretraining research, and points to TMax, an open recipe that trains Qwen models from 2B to 27B parameters into agents that work in a command-line terminal.

The talk ends on recursive self-improvement, the use of AI to automate AI research, and the students who ask him whether research still has a point. Lambert's answer is that language models speed up output far more than understanding. He says he barely writes code anymore yet is busier than ever as a scientist, while junior researchers may still need slow manual work to build understanding. He estimates that a project like RewardBench, his benchmark for reward models that took from October 2023 to March 2024, would now take a half to a quarter of the time, but says the organizationally complex work behind releases such as Tülu 3, Olmo 2 and Olmo 3 has changed far less.

Terms in this piece · Glossary
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
  • multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Read the source natolambert.com
More from Nathan Lambert
Recommended reads
Comments

Checking sign-in…

Loading comments…