OpenAI Says Two API Settings Nearly Tripled GPT-5.6 Sol’s ARC-AGI-3 Score

| | | | OpenAI's GPT-5.6 Sol just tripled its ARC-AGI-3 score using two API settings that also cut output tokens by 6x. | What You Need to Know: OpenAI detailed how retaining reasoning and applying context compaction in the Responses API lifted GPT-5.6 Sol’s public ARC-AGI-3 score 188% while slashing tokens. Simon Willison separately noted that GPT-5.6 optimizations already cut OpenAI’s serving costs 20%. Builders should test the same settings on long-horizon agent tasks this week. DEPTH OVER BREADTH (news items)Top Story OpenAI reported that GPT-5.6 Sol’s ARC-AGI-3 score rose 188% on the public set after switching to the Responses API, enabling retained reasoning, and turning on context compaction. The standard harness had been discarding reasoning after each move and dropping [...] OpenAI reported that GPT-5.6 Sol’s ARC-AGI-3 score rose 188% on the public set after switching to the Responses API, enabling retained reasoning, and turning on context compaction. The standard harness

With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies. The next improvement came from replacing rolling truncation with compaction⁠(opens in a new window), another setting in the Responses API. The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded. [...] GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture⁠(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%. Were 2D puzzle games unusually difficul

[]( GPT-5.6 Sol's ARC-AGI-3 scores jump dramatically when using OpenAI's custom harness with retained reasoning and compaction, compared to the official test harness. | Image: OpenAI OpenAI isn't using the official test environment, though. Instead, it runs GPT-5.6 Sol through its own Responses API with "Retained Reasoning," which keeps the model's chain of thought between steps, and "Compaction," which summarizes old context instead of truncating it. In the official harness, GPT-5.6 Sol scored just 7.8 percent because the model's reasoning gets discarded after each action. Ad [...] Subscribe now Source: OpenAI | via X wpDiscuz [...] He noted that ARC Prize has had "a lot of back and forth with OpenAI about how to best test their models, especially with regard to compaction," and welcomed the company "starting to figure out the answer." Different providers using different settings does create "a potential parity issue," Chollet said, but he considers that acceptable "as long as th

The original discrepancy involved several score figures. OpenAI noted that GPT-5.6 Sol had received a 7.8% overall ARC-AGI-3 score, while GPT-5.5 scored 0.4%. On the public task set used for its controlled comparison, GPT-5.6 Sol scored 13.3% with the official harness. OpenAI then rebuilt the evaluation through its Responses API. The revised harness retained reasoning between actions and replaced rolling truncation with compaction. According to the company's benchmark analysis, that configuration increased the public-set score to 38.3%. [...] OpenAI used the Responses API and passed the previous response identifier between turns. Its current model guidance explains that GPT-5.6 can reuse available reasoning items across turns. Developers can also control how much prior reasoning remains relevant through reasoning-context settings. According to OpenAI, retained reasoning changed GPT-5.6 Sol's behavior in two ways. The model spent less time thinking before each action, and it used more

API, enabling retained reasoning and turning on context compaction. The standard harness had been discarding reasoning after each move and dropping earlier actions as context filled, forcing the model to restart. With the new settings, the model builds on prior learning across multi-step two-dimensional game tasks that test adaptation to unfamiliar rules without instructions. The change also reduced output tokens by a factor of six. OpenAI recommends the same harness and settings it uses internally for any developer chasing frontier performance on agent style evals. The company further emphasized that benchmark scores reflect the model together with harness design and prompting choices [...] 1 comments ### Transcript: Good to have you on Models & Agents, Episode 126 for July 30th, 2026. Another day, another round of AI developments. Let's break it down. OpenAI's GPT 5.6 SoL just tripled its arc AGI3 score using two API settings that also cut output tokens by six times. OpenAI detailed

Log inSign up ## Post user avatar Tibo @thsottiaux Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason and work over multiple context windows with the help of our canonical compaction implementation. openai.com/index/how-two-… 11:30 PM · Jul 29, 20263MViews user avatar agent maxxer @claudeabuserr Jul 29 idrk what ur talking about but does this mean a reset? 55K user avatar Andrew Curran @AndrewCurran\_ Jul 29 The explanation was Momento. For anyone who skipped the blog. 51K user avatar Andrew Curran @AndrewCurran\_ Jul 29 With caffeine, and without caffeine. 26K # Join the conversation Read 658 more replies