Vibeleaderboard
← All Intel
Intel / video

Measuring Agents With Interactive Evaluations

Source
youtube.com
Author
OpenAI
Date

Context

Greg Kamradt, president of the ARC Prize Foundation, argues in this talk that measuring general intelligence requires interactive tasks, because one-shot question answering cannot test exploring an unfamiliar environment. The foundation follows François Chollet's definition of intelligence as how efficiently a system acquires new skills. Its planned ARC-AGI-3 , which Kamradt describes as a series of 150 original video games, gives no instructions: a player must click around, form hypotheses about the rules, and work out the goal.

Games are included only if a panel of members of the public, who have not seen them, can solve them, and official scoring uses a private set of unseen games so results reflect generalization. The foundation also counts how many actions human testers need and calls the resulting measure action efficiency: not just whether a player reaches the goal, but how directly. Kamradt presents this as a way to separate learning from brute-force trial and error. The talk's examples are illustrations, not reported benchmark results.

Terms in this piece · Glossary
  • Benchmark — A fixed set of tasks with a stated way of scoring them, used to compare models or setups.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
Read the source www.youtube.com
More from OpenAI
Recommended reads
Comments

Checking sign-in…

Loading comments…