Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
Source
Dwarkesh Patel
Author
Dwarkesh Patel
Date
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
A deep technical conversation with two Anthropic researchers on how far RL post-training can actually scale, where continual learning and inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → compute bottleneck AGI, and how mechanistic interpretability traces a model's reasoning — useful for engineers reasoning about AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → capabilities and near-term autonomy.
Key quotes
“Okay, so I think the biggest thing that's changed is that RL in language models has finally worked. We finally have proof of an algorithm that can give us expert human reliability and performance, given the right feedback loop.”
Sholto Douglas
“I really do think by the end of this year to this time next year, we will have software engineering agents that can do close to a day's worth of work for a junior engineer”
Sholto Douglas
“I actually think a Nobel Prize is more likely than a Pulitzer Prize-winning novel in some respects.”
Sholto Douglas
“You're going to spend literally weeks giving them feedback whereas we'll give up on a model in minutes.”
Dwarkesh Patel
“If you look at the circuit, you can see that it's not actually doing any of the math, it's paying attention to that you think the answer's four”
Trenton Bricken
Chapters
00:00:00How far can RL scale?
00:16:27Is continual learning a key bottleneck?
00:31:59Model self-awareness
00:50:32Taste and slop
01:00:51How soon to fully autonomous agents?
01:15:17Neuralese
01:18:55Inference compute will bottleneck AGI
01:23:01DeepSeek algorithmic improvements
01:37:42Why are LLMs 'baby AGI' but not AlphaZero?
