aiqbench takeAI Safety

The article analyzes a specific 2026 event where a human Go player defeated an advanced AI model with minimal handicap, challenging the prevailing assumption that current Go AIs represent a definitive benchmark for superintelligence. The author uses this discrepancy to question whether the perceived "superhuman" capabilities of these models are overstated or if they lack generalization skills required for real-world domains like software and strategy. This discussion centers on interpreting AI performance limits and their implications for future artificial general intelligence rather than reporting new hardware or algorithmic breakthroughs.

Source coveringLessWrong

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Human vs AI in Go in 2026: Lessons for superintelligence?

LessWrongOctober 19, 202616 min read

In July 2026, Korean Go player Shin Jin-seo defeated KataGo 2-1, with only[1] a 2-stone handicap (favoring Shin). I thought this was very surprising, and posted on LessWrong and Twitter asking how to interpret this. I didn’t get any entirely satisfactory answer, so I decided to spend ~one day looking into it and writing up my understanding. To preface this post: I don’t know all that much about Go and I'm a quite bad Go player (like, no better than 20 kyu or something like that). But in 2022 I did an interpretability project on a Go-playing model and have spent quite some time thinking about Go-playing AIs. Why I found this surprising Go is the canonical domain where AI models are considered to be vastly, vastly superhuman. We heard it from the best Go players themselves, who have faced off and been defeated by these Go-playing AIs. Ke Jie, the professional Go player then ranked world No. 1, said in 2017 after losing to AlphaGo that “AlphaGo to me is 100 percent perfection, to me AlphaGo is the god of the Go game”. I've been interested in Go-playing AIs partly because it serves as an intuition pump for me for superintelligence in other domains: Just imagine if an AI model became as superhuman in software/math/business strategy/persuasion as Go models in fact are. I specifically thought this of Go, and not necessarily chess or other games. It seems like many other people share this intuition. Noam Brown on Dwarkesh Podcast in September 2026 saying Go models are "orders of magnitude stronger than any human alive". And yet in 2026, a human defeated the best Go model, with only a 2-stone advantage. How is this possible? Do I have the wrong impression of just how superhuman Go models are? And if I’m wrong about this, are my beliefs about superintelligence therefore suspect? The case for “unimaginably, orders of magnitude stronger than any human alive” First of all, why did I believe that Go models are “unimaginably, orders of magnitude stronger than any human alive”? If I were to make the case for this, it would go something like this: AlphaGo Lee beat Lee Sedol, one of the top Go players at the time, 4-1 in 2016 One year later, AlphaGo Zero was already beating AlphaGo Lee 100-0 In 2020, the creator of KataGo commented that KataGo was "very likely" to be "much stronger than AlphaGo Lee" and also likely stronger than AlphaGo Zero And KataGo has very much continued to improve since then. In case you were not familiar with KataGo: It started in 2017 as a personal project of lightvector (David Wu), a Go enthusiast at Jane Street, and eventually became an open source, community-led distributed training project. Its training run is still ongoing, and the last update on their website at the time of writing is from September 15, 2026. It has a self-play Elo of over 14,000—of course, self-play Elo is not the same as actual Elo, not even close—nonetheless, KataGo has been in self-play training and improving, at some rate, for nearly a decade. It is this model that Shin Jin-seo faced, and defeated… A chart of KataGo's self-play Elo over time. Just how large is a 2-stone handicap? … with a 2-stone handicap. Is a 2-stone handicap a lot? Apparently it is, and it seems to satisfy most people as the explanation for Shin's victory. The top comment on the Hacker News thread says: 2 stones is historically the gap between a 9P ranked and a 1P ranked professional player (very roughly the gap between super grandmasters and an almost grandmaster) A comment I got on my LessWrong short form on this was: 2 stones is a substantial handicap at high levels. The saying I recall from the pre-Go-AI days was that a Go grandmaster would take a three-stone handicap to play against God. I tried to hunt this saying down. With Opus 5.5’s help, I found the Hand of God page on Sensei’s Library, which says: A quote from Bruce Wilcox's book EZ-GO: "People have asked how much stronger than a professional player is God? While the match has yet to be played, most estimates place God three ranks above top professionals. One pro is reported to have said he wouldn't bet his life against God without a four-stone handicap. [emphasis mine]" By the way, I couldn't find the content of the actual book on the internet, but while searching, I discovered this is what the cover looks like… A totally ordinary Go strategy book. We also know that ranks and stones are both extremely nonlinear measures of strength, especially at the top. But in any case, the gap between God and a pro seems to be approximately 4 stones, or 3 ranks. So maybe 2 stones is such a huge handicap that renders Shin’s victory not at all remarkable? I don’t think so. Also on the Hand of God page is this commentary, written in 2020: After the surprising (to say the least) victory of AlphaGo in 2016, those fanciful estimations of the strength of perfect play came again in a new light ; even then, no one could have predicted than a mere four years later, the best computers would easily

To preface this post: I don’t know all that much about Go and I'm a quite bad Go player (like, no better than 20 kyu or something like that). But in 2022 I did an interpretability project on a Go-playing model and have spent quite some time thinking about Go-playing AIs.

Go is the canonical domain where AI models are considered to be vastly, vastly superhuman. We heard it from the best Go players themselves, who have faced off and been defeated by these Go-playing AIs. Ke Jie, the professional Go player then ranked world No. 1, said in 2017 after losing to AlphaGo that “AlphaGo to me is 100 percent perfection, to me AlphaGo is the god of the Go game”.

I've been interested in Go-playing AIs partly because it serves as an intuition pump for me for superintelligence in other domains: Just imagine if an AI model became as superhuman in software/math/business strategy/persuasion as Go models in fact are. I specifically thought this of Go, and not necessarily chess or other games. It seems like many other people share this intuition.

Noam Brown on Dwarkesh Podcast in September 2026 saying Go models are "orders of magnitude stronger than any human alive".

And yet in 2026, a human defeated the best Go model, with only a 2-stone advantage. How is this possible? Do I have the wrong impression of just how superhuman Go models are?

And if I’m wrong about this, are my beliefs about superintelligence therefore suspect?

First of all, why did I believe that Go models are “unimaginably, orders of magnitude stronger than any human alive”? If I were to make the case for this, it would go something like this:

And KataGo has very much continued to improve since then. In case you were not familiar with KataGo: It started in 2017 as a personal project of lightvector (David Wu), a Go enthusiast at Jane Street, and eventually became an open source, community-led distributed training project. Its training run is still ongoing, and the last update on their website at the time of writing is from September 15, 2026. It has a self-play Elo of over 14,000—of course, self-play Elo is not the same as actual Elo, not even close—nonetheless, KataGo has been in self-play training and improving, at some rate, for nearly a decade.

A chart of KataGo's self-play Elo over time.

Just how large is a 2-stone handicap?

… with a 2-stone handicap.

Is a 2-stone handicap a lot? Apparently it is, and it seems to satisfy most people as the explanation for Shin's victory. The top comment on the Hacker News thread says:

2 stones is a substantial handicap at high levels. The saying I recall from the pre-Go-AI days was that a Go grandmaster would take a three-stone handicap to play against God.

I tried to hunt this saying down. With Opus 5.5’s help, I found the Hand of God page on Sensei’s Library, which says:

A quote from Bruce Wilcox's book EZ-GO: "People have asked how much stronger than a professional player is God? While the match has yet to be played, most estimates place God three ranks above top professionals. One pro is reported to have said he wouldn't bet his life against God without a four-stone handicap. [emphasis mine]"

A totally ordinary Go strategy book.

We also know that ranks and stones are both extremely nonlinear measures of strength, especially at the top. But in any case, the gap between God and a pro seems to be approximately 4 stones, or 3 ranks. So maybe 2 stones is such a huge handicap that renders Shin’s victory not at all remarkable?

I don’t think so. Also on the Hand of God page is this commentary, written in 2020:

Opus 5.5 thinks that this comment most likely refers to the series of matches between FineArt (a proprietary Go model from Tencent) and top pros in 2018 where they played “casual” games with FineArt at a 2-stone handicap. FineArt won about 90 percent of the time, including winning over Ke Jie, the top Go player at the time. The games were “casual” because each turn had a 1-minute timer, unlike “normal” Go games.

My research also found this prediction market on whether a Go AI can beat a top human pro at a 5-stone handicap, which was resolved YES in 2024.

So, a 2-stone handicap is a lot, but I don’t think that makes Shin’s victory completely unsurprising. Plus, in Game 3, Shin won by 11.5 points—since a 2-stone handicap is about equivalent to a 18.5 point advantage at the start, it means that he only "lost" 7 points throughout the game—an amount that would have put him at about even with KataGo in a game with no komi and no handicap.

So, what happened?

After one day of research, my best understanding is the combination of these three factors: (1) Shin is an incredible, historical Go player who is also trained in the age of AI. (2) The configurations used in the matches were likely not very favorable to KataGo. (3) Shin knew his opponent extremely well, and KataGo did not.

(1) The first thing to understand here is simply that Shin Jin-seo is the greatest Go player who’s ever lived. American Go teacher and streamer Nate Morse says in his interview with Frank Lantz on this match that “Shin Jin-seo is the number one player in the world by a significant margin. He’s more dominant over Go than Magnus Carlsen is over chess, and I don’t think it’s close.” He is also among the first Go professionals that was “brought up” on AI tools. To quote the Korean outlet KMJ, “Shin Jin-seo embodies the first true AI-native Go champion. His rise to world No. 1 was built on AI-driven study methods, including pattern exploration, self-directed simulations, and data-rich analysis tools.” In any case, Shin is incredibly strong. According to gennan’s estimate on Reddit, Shin was slightly above AlphaGo Lee in Elo in 2019, and redditors think he would be able to (easily) win over AlphaGo Lee in 2026.

(2) On the KataGo side, the setting that KataGo ran on was poorly documented, but believed/suspected to be a relatively weak one. Morse says that “KataGo was not as aggressive as it’s supposed to be for a handicap game, because some of those features were tuned down a little bit for KataGo this first time” with no further explanation. He also thought that KataGo made some very unusual moves across the games, especially move 3 of Game 1. There was also some heated discussion about this on the Computer Go Discord during the Shin vs KataGo matches. Some users argued the organizers’ KataGo played like a low-visit engine, not the full-strength 4×3090 setup claimed, but others thought the evidence was insufficient to prove anything. Regardless, KataGo was not using “pondering”, which meant that it was only allowed about 20 seconds of thinking time for each move, and was simply sitting idle while Shin was thinking. In other words, KataGo was definitely not configured to be as strong as it can be.

(3) Shin very much optimized for the objective of beating KataGo, in a way akin to “metagaming”. He spent months studying KataGo before the competition. In the Morse interview, he and Lantz discussed the “space-filling” joseki in the lower right corner that Shin played in Game 2, and why it’s advantageous to the handicapped player. KataGo, on the other hand, did not “know” that its opponent was a much weaker, human player. To my best understanding, KataGo likely did know that it was playing a handicapped game, but only slightly[2]. It did not fully “know” that it was facing a much weaker opponent, and did not use the more optimal strategy for such games of taking much more risks. Shin himself commented on this in his interview with Korea Times: "AI's weakness seems to be that it is too perfect. Even when it is behind, it doesn't take desperate risks." Of course, even with the "correct" handicap settings, KataGo still wouldn't have the ability to model a human opponent—the handicap mode was trained by having a high-playout KataGo play against a low-playout copy of itself.

Lastly, since this has come up in talking to a few people about this—as far as I can tell, Shin did not win by exploiting adversarial examples or known mistakes of KataGo, such as the cyclic group attack in Wang et al. (2022).

Since the exact settings and therefore strength of the KataGo in these games are unknown, pretty big error bars remain in my mind. But I have a few takeaways.

I will no longer think of Go-playing AIs as unimaginably superhuman, or orders of magnitude stronger than the best human players. Go-playing AIs are very superhuman, but apparently users on the Computer Go Discord can still find mistakes in KataGo[3], top pros are getting better at Go by learning from the AIs, and now we have seen a human come within 2 stones of KataGo. I don’t think this is what it means to be orders of magnitude ahead.

Actually, it seems that this update doesn’t even rest much upon Shin’s victory over KataGo, and follows just from the “God is four stones away from top pros” measure. Go is a game such that with a finite amount of handicap, even God (i.e. perfect play) cannot always win. I do think that the superintelligence discourse sometimes pull the card that “well, a superintelligence can win over anything”. That’s just not true in Go, and maybe in other domains too.

In hindsight, it seems that we are just not very good at assessing how superhuman the Go AIs are. Ke Jie thought AlphaGo was God-level already in 2017. Kind of makes sense: how shall we know, once the AI has surpassed the understanding of our best human? With the notion of handicaps, in Go, it seems like it should have been easy to quantify the level of superhuman-ness, but we seemed in practice still quite confused. This makes me think that understanding general superintelligence is going to be even trickier.

A meme on superintelligence that I last saw featured in Marginal Revolution about the TheoPhysEval results.

I asked Claude to "make a new version of this meme that reflects what I'm trying to argue in this piece."

I am slightly confused why KataGo has not achieved perfect play. Morse thinks that KataGo is within one stone away from perfect play, but mistakes are still regularly discovered in KataGo. (He also says that there is no mystery in the mistakes: if you then train KataGo in those situations where it was making a mistake, the mistake goes away.) Yet somehow RL and self-play has not rooted out all of the mistakes that us mere humans are able to find. Why is this? Does this imply RL+self-play is less robust as an algorithm than we thought? I think the simple answer here is probably scaling—Go models are tiny by 2026 standards—but I’m not sure whether this is all there is.

And lastly, could the meta-game as important or more than the game? Shin's victory seems to at least partially hinge not on Shin knowing what moves are better, but that he's able to optimize for the whole "facing off superintelligent AI" situation better. What does this say about our future with other superintelligences?

^More on this later.