ARC-AGI
arcprize.org- Category
- Developer Tools
- Rank
- No. 1087Tools index
Previous survey · No. 1046 ·
- Listed in
- #30 Find AI benchmarks
- Type
- TOOL
- Builder
- arcprize
- GitHub
- 76 stars
- Latest release
- v0.9.6
- Date
About
Measures skill acquisition and generalization on novel abstract tasks, including interactive agentic environments in ARC-AGI-3.
What it does
ARC-AGI is a Python toolkit for running game-like environments through one interaction model. Agents receive frames and available actions, submit moves, reset sessions, and track results through scorecards. Environments can run from local Python sources or through a remote service, with terminal, graphical, and custom rendering options.
Why it's ranked here
Its value comes from covering the full experiment loop without forcing one execution setup. The same toolkit discovers environments, handles local and remote sessions, records actions, renders frames, and aggregates scorecards. A REST server also opens the interaction model to non-Python clients. The narrow Python requirement and competition restrictions temper that utility.
What's good
Local, online, offline, and competition modes make deployment choices explicit. Scorecards track environments, completed levels, actions, resets, and per-level results. Optional JSONL recordings can include frames and reasoning. Terminal, fast terminal, graphical, and custom renderers support both automated runs and human inspection. Typed models validate environment metadata and API responses.
Tradeoffs
The package requires Python 3.12 or newer and brings Flask, Matplotlib, Pillow, Pydantic, Requests, and the separate engine package. Normal operation may contact the hosted service to obtain an anonymous key and discover games. Local environments execute dynamically loaded Python source. Competition mode permits one session per environment, one scorecard, level-only resets, and no score inspection while running.
How to use it well
Use it for building and comparing agents that need repeatable environment access, action loops, run recordings, and consolidated scoring. Start with terminal rendering, attach reasoning metadata when useful, save JSONL for later inspection, and switch between offline and remote execution deliberately. It does not supply an agent strategy or training system. The included examples select random actions.
Technical notes+
pyproject.toml defines the arc-agi package with a uv_build backend, Python >=3.12, typed package data, strict mypy settings, and runtime dependencies including arcengine, Flask, Matplotlib, Pydantic, Pillow, and Requests. arc_agi/base.py implements environment discovery and the NORMAL, ONLINE, OFFLINE, and COMPETITION modes. arc_agi/local_wrapper.py reads game source and executes it in a generated module, caching validated ARCBaseGame subclasses. arc_agi/remote_wrapper.py maps resets and actions to HTTP requests and converts validated responses into NumPy-backed frames. arc_agi/server.py registers the Flask REST routes, while arc_agi/wrapper.py handles JSONL recording, renderer callbacks, action exposure, and scorecard updates. arc_agi/scorecard.py weights completed levels by level index, squares baseline-to-action efficiency, and caps individual level scores.
Observed
- Primary language
- Python
- Runtime requirement
- Python 3.12 or newer
- Packaging
- Installable as arc-agi with pip or uv; built with uv_build
- Interfaces
- Python library and Flask REST API server
- Execution modes
- Local plus remote, online-only, offline-only, and competition modes
- Output surfaces
- Terminal, fast terminal, Matplotlib, custom rendering, and optional JSONL recordings
- License declaration
- Package metadata points to a repository LICENSE file
Read from README.md, pyproject.toml, main.py, quickstart.py, arc_agi/api.py, arc_agi/base.py, arc_agi/server.py, arc_agi/models.py, arc_agi/wrapper.py, arc_agi/__init__.py, arc_agi/rendering.py, arc_agi/scorecard.py, arc_agi/local_wrapper.py, arc_agi/remote_wrapper.py.
Intel on ARC-AGI
- NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents
- ARC-AGI-3 blown open in hours, and the benchmark politics that followed
- Two harness settings put GPT-5.6 Sol atop ARC-AGI-3
- Two settings tripled GPT-5.6 Sol’s ARC-AGI-3 score
- Gemini 3.1 Pro more than doubles its predecessor's ARC-AGI-2 score
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.