Finishing a puzzle does not necessarily show that an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → understood it. In ARC Prize's first month of testing its interactive benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →, random exploration could eventually complete some games. That made success alone a weak way to distinguish learning from repeated trial and error.
The preview report therefore examines how many actions an agent needs, separating exploration from execution and comparing performance with human baselines. Those baselines also need care: a person's first authenticated attempt might follow earlier anonymous practice. The report describes plans for controlled human testing; its public play data should not be mistaken for that standardized comparison.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
