ARCgentica

Why it's ranked here

The result is the reason: the authors report 85.28% on the ARC-AGI-2 public evaluation set with Claude Opus 4.6 at $6.94 per task, and they committed the logs of that run plus a command to re-grade it. The code is small, MIT licensed and pinned to exact dependency versions, so the claim can be audited rather than taken on trust. What limits it is scope: it solves one benchmark and nothing else, and reproducing the headline number takes real money and roughly 25 minutes per task on average.

What's good

The design is short enough to read in an afternoon. Grading is exact match on the whole grid, with a softer cell-by-cell measure the models use while they iterate, and the final tally follows the competition rule of two guesses per test input. Answers are code, so every prediction can be rerun and inspected. Each run writes its settings, every attempt's result and every agent's transcript to disk, and an interrupted run resumes by ID, skipping puzzles already done. Delegation is capped at ten agents per attempt, counted across the whole tree, so a runaway chain of helpers hits a hard wall.

Tradeoffs

The recommended server launch turns sandboxing off, and generated programs run as ordinary child processes with your user's permissions, so model-written code executes on your machine. Retries fire only on one specific OpenAI content-policy error; other failures are recorded and not retried, although the documentation describes retries for transient errors. The documented default model differs from the one the code actually defaults to, and the server launch command is missing a line continuation, so the port flag never reaches the server. Python is pinned to exactly 3.12.11. The reported run used high reasoning effort, not the extra-high default. Continuous integration runs formatting and lint checks only.

How to use it well

Treat it as a reference implementation if you study program synthesis or multi-agent orchestration, or as the baseline to beat on ARC. Start by re-grading the committed run, which costs nothing, then try ten puzzles before a full pass: at the reported averages each task costs about seven dollars and takes about 25 minutes. Run the agent server in a container or virtual machine, since sandboxing is off, and fix the broken line in its launch command. It is not a general agent framework; for that, look at the Agentica SDK it is built on.

Observed

License

MIT (LICENSE)

Language

Python, pinned to exactly 3.12.11 in pyproject.toml

Packaging

uv-managed project run from a clone; pyproject.toml declares no console entry point

Interface

Command-line runner with argparse flags (main.py); requires a separately cloned Agentica server

Model providers

OpenAI, Anthropic or OpenRouter through the Agentica server

Selectable models

Three hardcoded choices: openai/gpt-5.2, anthropic/claude-opus-4-5, anthropic/claude-opus-4-6

Code execution

Generated programs run in a plain subprocess with a 5 second default timeout; README.md server command sets sandbox mode to no_sandbox

Scoring

Exact grid match on the first two non-empty outputs per test input (score.py)

Reported run

Logs and configuration of the reported run are committed (output/2025/anthropic/claude-opus-4-6/final/config.json)

Continuous integration

ruff format and ruff check only, no test step (.github/workflows/ci.yml)