← All IntelClip / EntertainmentLLMs automate what you can verify
From Andrej Karpathy on Vibe Coding & Agentic Engineering (AI Ascent 2026) · ≈10:03
“traditional computers can easily automate what you can specify in code. And uh, kind of this latest round of LLMs can easily automate what you can uh, verify in a certain in a certain sense.”
“they end up basically uh, progressing and creating these like jagged entities that really peak in capability in kind of like verifiable domains like math and code and adjacent.”
What’s in it
- Why LLMs excel at code and math but stay jagged elsewhere
- How RL verification rewards shape frontier model capabilities
- Which verifiable domains get prioritized based on economic value
Clip transcript
verifiable? Uh, yes, so I I spent uh, some time writing about verifiability and um, basically like traditional computers can easily automate what you can specify in code. And uh, kind of this latest round of LLMs can easily automate what you can uh, verify in a certain in a certain sense. Uh, because the way this works is that when frontier labs are training these LLMs, these are giant reinforcement learning environments. So, they are given a verification rewards. And then because of the way that these models are trained, they end up basically uh, progressing and creating these like jagged entities that really peak in capability in kind of like verifiable domains like math and code and adjacent. And kind of like stagnate and are a little bit um, you know, rougher on the edges when uh, things are not kind of like in that in that space. So, I think the reason I wrote about verifiability is I'm trying to understand why these things are so jagged. Um, and some of it has to do with how the labs train the models, but I think some of it also has to do with um, the focus of the labs and what they happen to put into the data distribution. Uh, because some things basically are significantly more valuable in economy and end up creating more environments because the labs wanted to work in those settings. So, I think code is a good example of that. There's probably lots of verifiable environments they could think about that happen not to make it into the mix because they're just not that useful to have the capability around. Um,
Comments
Sign in to comment.
Loading comments…