DeepMind's Kavukcuoglu: No Single Benchmark Will Signal AGI
- Source
- video.twimg.com
- Author
- GoogleDeepMind
- Date
Context
Koray Kavukcuoglu, the senior vice president who steers development of Google DeepMind's Gemini models, argues that no will ever announce the arrival of artificial general intelligence, or AGI, and that progress should be judged by how far people come to trust AI models with real work. Google DeepMind's post promoting his podcast conversation with the host Hannah Fry puts his view in one line: there won't be a single test that tells us we've reached AGI, and progress instead looks like models becoming less spiky and more reliable across daily life.
In the part of the interview that was transcribed, he grounds that view in the field's history. For 15 or 20 years, he says, researchers tried to define AGI through benchmarks, sets of test questions or tasks with a score, expecting that passing them would mean a model had achieved some kind of intelligence. "We always achieved those benchmarks," he says, and each time researchers felt they had not captured the gist of intelligence. His alternative is usefulness and trust: the more people rely on a model to make decisions for them or with them, the more it will start to feel like AGI. That is why he calls products, the apps and tools people actually use, the path to it. Products show what people need AI for, and he doubts AGI can be built without them.
Spiky is his word for uneven ability: today's models do some things very well and others badly. He says they are "much less spiky than what they were, but still they're spiky." When Fry raises everyday jobs such as booking a flight or a restaurant, he treats them as the test case. A model that can solve hard problems but cannot book a restaurant is, in his words, the definition of spikiness, and a product forces a lab to fix all of those weak spots rather than choose among them. "We are not coming up with benchmarks that define intelligence," he says. "We are trying to build it."
The transcript this explanation draws on has a gap that matters for the headline. The episode's chapter list puts a segment on safety and agent teams at 30:50 and one titled Benchmarks at 40:35, but speech was transcribed only for the first 30 minutes. What he says in the dedicated benchmarks segment is therefore not covered. The points above come from the earlier chapter on products. In a later chapter on recursive self-improvement, the idea of AI systems carrying out the research that produces their own successors, he adds that no single benchmark defines what you are actually aiming for. In that transcribed part, he describes trust as the yardstick without saying how it would be measured. Google DeepMind researchers proposed one way to measure uneven ability in a March framework that would place a model's results, ability by ability, against human results on the same tasks, though Kavukcuoglu does not refer to it in the transcribed portion.
Checking sign-in…
Loading comments…






