OpenAI says two API settings nearly tripled GPT-5.6 Sol’s ARC-AGI-3 score

#### Foundation #### Leaderboards #### Benchmark #### Prize #### Research #### Content ARC Prize Verified # GPT-5.6 Sol GPT-5.6 Sol is the standout model of the GPT-5.6 family. Sol at max reasoning effort is the only performant model (as of July 2026) averaging 13.33% on Public and 7.78% on Semi-Private. It is the first model to win an ARC-AGI-3 public game (ft09, 87%). Sol is able to read an unfamiliar scene correctly and in the game's own vocabulary. It treats a failed hypothesis as a reason to re-plan rather than thrash. Most agent failures are upstream of the code they write or the action they take. Sol is able to perform on ARC-AGI not because it executes better, but because it correctly orients itself in a new environment first. Related models: GPT-5.6 Terra, GPT-5.6 Luna [...] Related models: GPT-5.6 Terra, GPT-5.6 Luna ## ARC-AGI 3 leaderboard ## Verified scores | Variant | ARC-AGI-1 | ARC-AGI-2 | ARC-AGI-3 | --- --- | | Max | 96.5% | 92.5% | 7.78% | | Extra High

parent | next (javascript:void(0)) Gemini models struggle with agentic coding/tool use/exploration, but they are actually quite smart in one-shot reasoning. They're not as far behind as people think. It's mostly post-training and productization issues, which are easier to fix than pre-training/mid-training issues. Agentic coding data: | | | | | | | --- | | | monk\_grilla 18 days ago | prev | next (javascript:void(0)) This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"? Replay of Sol attempting the game: It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't [...] | | | | | --- | | | andriy\_koval 18 days ago | parent | next (javascript:void(0)) > Can someone explain how it in any way indicates how close we are to "AGI"? I think it is historical

GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture⁠(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%. Were 2D puzzle games unusually difficult for our models? Or was something else going on? Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set. GPT-5.6 Sol on the ARC-AGI-3 Public Set [...] Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair perfo

ActivityExploreProfile ### Make money doing the work you believe in Think AI 3h @thinktankai GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. ``` We found that enabling two API settings tripled our scores with 6x fewer output tokens. ``` Jul 30 at 7:00 AM Relevant people #### Log in or sign up Join the most interesting and insightful discussions.

“out of distribution.” AI is incredibly adept at solving problems in-distribution, but the ones outside—those it struggles with. [...] I’m referring to GPT-5.6’s score on the benchmark known as ARC-AGI-3—a personal favorite of mine—which amounts to the scandalous figure of 7.8%. (From here on, I only refer to GPT-5.6 Sol, the biggest of the three versions of this model, which include Terra and Luna; Read the full blog post on ARC-AGI-3 here.) It’s scandalous for three reasons. [...] It’s scandalous for three reasons. First, because it is not, in fact, scandalous at all. ARC-AGI-3 is a benchmark designed to look easy to the human eye while requiring a type of intelligence—”fluid, not crystallized,” as its creator, François Chollet, likes to put it—that AI struggles with. If you try it yourself, it might take a little while to get the hang of it, but soon enough you start to grasp the idea of each game, the patterns of colors and movements, etc. That’s why 7.8% is terribly low; humans

Log inSign up ## Post user avatar GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with 6x fewer output tokens. 00:00 11:57 PM · Jul 29, 20261.1MViews user avatar OpenAI @OpenAI Jul 29 ARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting over. 00:00 100K user avatar OpenAI @OpenAI Jul 29 [...] 100K user avatar OpenAI @OpenAI Jul 29 We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tok