François Chollet: How We Get To AGI

François Chollet argues that the ARC puzzle benchmark shows why scaling up pre-training did not produce flexible intelligence. In a 2025 Y Combinator talk, Chollet describes ARC, a benchmark released in 2019, as small puzzles that each must be figured out fresh, so memorised patterns do not help.

The evidence Chollet cites is that base language models scaled up roughly 50,000 times scored about 10 percent on the first version, while anyone in the room would score well above 95 percent. Chollet says progress came when systems began adapting at test time, changing their behaviour for the specific task while solving it. An ARC-tuned version of OpenAI's o3 preview is cited as the first to show human-level performance on that version.

The first version then saturated, Chollet says, and a harder second version followed in March 2025. By that account base models score 0 percent on it and single-chain reasoning systems 1 to 2 percent, while about 400 people tested in person solved every task between them. These figures are Chollet's own.