Measuring progress toward AGI: A cognitive framework
- Source
- deepmind.google
- Author
- Google DeepMind
- Date
Context
Google DeepMind proposes measuring progress toward general AI through a profile of cognitive abilities compared with human performance. Its framework separates abilities such as learning, , reasoning, and monitoring one's own thinking, so strength in one area need not conceal weakness in another.
The proposed procedure gives AI systems held-out tasks, collects results from a demographically representative sample of adults on the same tasks, and places the AI results within those human distributions. That would make comparisons more specific than a single claim of human-level intelligence. The article presents an design and identifies missing tests; it does not report that a model has passed the proposed assessment.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
“But it can be difficult to know how close we are to this key milestone, because there’s a lack of empirical tools for evaluating systems’ general intelligence.”
Google DeepMind
“Our framework draws on decades of research from psychology, neuroscience and cognitive science to develop a cognitive taxonomy. It identifies 10 key cognitive abilities that we hypothesize will be important for general intelligence in AI systems”
Google DeepMind
“To understand AI capabilities across these cognitive abilities, we propose a three-stage evaluation protocol that benchmarks system performance in relation to human capabilities”
Google DeepMind
“The hackathon encourages the community to design evaluations for five cognitive abilities where the evaluation gap is the largest: learning, metacognition, attention, executive functions and social cognition.”
Google DeepMind
“We are offering a total prize pool of $200,000: $10,000 awards for the top two submissions in each of the five tracks, and $25,000 grand prizes for the four absolute best overall submissions.”
Google DeepMind
Checking sign-in…
Loading comments…