ARC Prize
22 Intel
ARC Prize is a nonprofit organization that develops AI benchmarks and runs competitions to advance open research in artificial general intelligence. François Chollet and Mike Knoop co-founded the organization; Greg Kamradt serves as president.
Is this you? Sign in with X to claim this profile.
Intel
ARC Prize runs 2.8K LLM-vs-LLM snake games across 50 models. Reasoning models like o3-mini and DeepSeek won 78% of matches, and results highlight how much board context models need to play well.
ARC Prize studied 160 replays and reasoning traces of GPT-5.5 and Opus 4.7 on ARC-AGI-3 and identified three failure modes in hypothesis formation, abstraction and learning transfer, releasing its analysis package.
ARC Prize launches ARC-AGI-2, a harder benchmark for AI reasoning systems that stays easy for humans, alongside ARC Prize 2025. It frames o3-preview as a zero-to-one step and explains the efficiency gap that remains.
ARC Prize evaluates o3 and o4-mini on ARC-AGI, reporting scores by reasoning setting and explaining why incomplete high-compute completions made some results unreportable.
ARC Prize reproduces HRM's 32% ARC-AGI-1 and 2% ARC-AGI-2 scores on semi-private sets, then runs ablations suggesting the headline brain-inspired architecture is not the main driver of performance.
ARC Prize tests OpenAI's o1 models on ARC-AGI-Pub with its standard harness. It examines chain-of-thought at training and inference time and why strong competition-math results coexist with modest ARC scores.
ARC Prize reports GPT-6 Astra at 62.7% on ARC-AGI-3 semi-private for $26K, beating median human action efficiency on 96% of levels by building compact symbolic world models.
ARC Prize introduces ARC-AGI-Pub, a public-eval leaderboard that lifts compute and internet restrictions so frontier APIs can be compared, backed by a $150,000 verification fund.
ARC Prize reflects on frontier reasoning systems six months after the o1 breakthrough, separating long-running inference and other test-time adaptation methods and weighing the claim that test-time compute alone gets to AGI.
ARC Prize scores DeepSeek's R1-Zero and R1 against o1 on ARC-AGI-1, finding all three near 15-20%. It argues the RL-only R1-Zero is the more important result and discusses what that implies for scaling narratives.
ARC Prize reports OpenAI's o3-preview scoring 75.7% on ARC-AGI-1 at the $10k compute limit and 87.5% at high compute. The post covers the testing setup, cost, and why the result matters for assumptions about capability.
ARC Prize 2025 results: 1,455 teams, a Kaggle SOTA of 24% at $0.20 per task, Opus 4.5 at 37.6%, and a Poetiq refinement solution on Gemini 3 Pro at 54%, with analysis of the refinement-loop trend.
ARC Prize announces its first ARC-AGI-3 milestone prize winners, led by a small open model that treats each game as an interactive coding problem through a Python REPL.
ARC Prize reports 30-day lessons from ARC-AGI-3 preview games: humans solve them readily, agents struggle, some games permitted random search, and action efficiency separates human-level from AI-level performance.
ARC Prize announces 2024 winners and a technical report. The SOTA moved from 33% to 55.5%, the ARChitects won with 53.5%, and the post compares Kaggle results against ARC-AGI-Pub frontier-model approaches.
ARC Prize launches a verification program that certifies ARC-AGI scores on official hidden datasets, with an independent academic panel auditing and open-sourcing the testing process.
The ARC Prize Foundation's testing policy for verified ARC-AGI results, covering governance and funding independence, evaluation methodology, dataset security, provider handling, and publication and reproducibility rules.
ARC Prize publishes the human dataset for ARC-AGI-3 from a 458-participant study across diverse backgrounds, establishing baselines for how people learn novel interactive environments.
ARC Prize launches ARC-AGI-3: hundreds of handcrafted interactive game environments with no instructions or stated goals. Humans score 100%, frontier AI 0.51%. ARC Prize 2026 offers over $2M in prizes.
Walks through the top ARC-AGI-Pub submissions of 2024, including Jeremy Berman's evolutionary test-time compute, where Claude generates and mutates Python transform functions. Both approaches are open source and reproducible.
ARC Prize Foundation's technical report for ARC-AGI-2, covering benchmark design improvements, human performance testing, current state-of-the-art scores, and the associated competition.
Mike Knoop and François Chollet argue that deep learning alone cannot solve ARC-AGI and sketch a hybrid with program synthesis. The post covers compute scaling history, discrete program search, and candidate approaches.