Two settings tripled GPT-5.6 Sol’s ARC-AGI-3 score
- Source
- openai.com
- Author
- OpenAI
- Date
Context
OpenAI reports roughly tripling GPT-5.6 Sol's score on ARC-AGI-3's public tasks while using one-sixth as many output in a changed evaluation setup. Its experiment retained the model's reasoning between game actions and summarized long histories instead of repeatedly discarding older messages.
Those changes let the carry plans and observations forward while learning unfamiliar games. The comparison also used different -limit units, so it measures the changed setup as a whole. The result demonstrates how the software managing an agent's memory can affect a reasoning benchmark's score.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
“turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set”
OpenAI
“Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.”
OpenAI
“on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%”
OpenAI
“This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking.”
OpenAI
“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”
OpenAI
Checking sign-in…
Loading comments…