Measuring Agents With Interactive Evaluations
- Source
- youtube.com
- Author
- OpenAI
- Date
Context
Greg Kamradt, president of the ARC Prize Foundation, argues in this talk that measuring general intelligence requires interactive tasks, because one-shot question answering cannot test exploring an unfamiliar environment. The foundation follows François Chollet's definition of intelligence as how efficiently a system acquires new skills. Its planned ARC-AGI-3 , which Kamradt describes as a series of 150 original video games, gives no instructions: a player must click around, form hypotheses about the rules, and work out the goal.
Games are included only if a panel of members of the public, who have not seen them, can solve them, and official scoring uses a private set of unseen games so results reflect generalization. The foundation also counts how many actions human testers need and calls the resulting measure action efficiency: not just whether a player reaches the goal, but how directly. Kamradt presents this as a way to separate learning from brute-force trial and error. The talk's examples are illustrations, not reported benchmark results.
Checking sign-in…
Loading comments…






