Two settings tripled GPT-5.6 Sol’s ARC-AGI-3 score

Terms in this piece · Glossary

context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Why it matters

The result is a clean warning against reading scores as model-only measurements: memory, management and design can dominate the apparent capability.

Key quotes

“turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set”
OpenAI

“Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.”
OpenAI

“on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%”
OpenAI

“This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking.”
OpenAI

“Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.”
OpenAI

Intel