Two harness settings put GPT-5.6 Sol atop ARC-AGI-3
- Source
- x.com
- Date
goblin-level blog post https://t.co/pmcqmDjE4f
Context
OpenAI's ARC-AGI-3 experiment shows how the software running a model can change its result. The benchmark asks agents to work out unfamiliar interactive games. On the benchmark's public task set, the company reports that Sol's score rose from 13.3% to 38.3% after changing how its reasoning and history were retained.
The original setup discarded private reasoning after each action and eventually dropped older history. The revised setup preserved reasoning and summarized accumulated so earlier discoveries remained useful. The related Intel coverage therefore supports a conclusion about the whole evaluation setup, rather than an unexplained jump in the underlying model's ability.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
Checking sign-in…
Loading comments…