The data black hole at the center of AI

An AI model can acquire more skills without becoming substantially better at learning from a few new examples. That is the distinction Dwarkesh Patel argues for in his essay on sample efficiency. He attributes much recent progress to expanding and improving training data, including expert demonstrations and practice tasks whose answers can be checked.

Patel also explains why expensive training can still make economic sense: a skill learned once can be reused across enormous numbers of sessions. His concern is work that repeatedly presents unfamiliar, situation-specific problems. The essay asks how far preparing skills in advance can carry automation, while treating more efficient learning from experience as an unresolved capability.

“We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data.”

Dwarkesh Patel

“We are building some Frankenstein’s monster, with a billion grafts of carefully constructed examples sewn together.”

Dwarkesh Patel

“Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy.”

Dwarkesh Patel

“Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file.”

Dwarkesh Patel

“I would be willing to bet that there’s overall more demand for human software engineers in 2028 than there is now, largely due to the complementary input of AI.”

Dwarkesh Patel