AlphaZero — Mastering Chess, Shogi and Go from the Rules Alone

AlphaZero is a single algorithm from DeepMind that learned chess, shogi and Go purely through reinforcement learning from self-play, given no knowledge beyond the rules of each game. The paper, "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm", was posted to arXiv on 5 December 2017 and reported wins over the strongest program in each field: Stockfish in chess, Elmo in shogi and AlphaGo Zero in Go. After criticism of the match conditions, the full evaluation was published in Science on 7 December 2018 (volume 362, issue 6419, pages 1140–1144).

The story of AlphaGo, which beat Lee Sedol, and of AlphaGo Zero, which dropped human games, is told on its own page. What was new about AlphaZero was that one method, one network architecture and one set of hyperparameters learned three quite different games.

What was new

As the preprint's abstract puts it, the strongest chess programs combined sophisticated search techniques, domain-specific adaptations and handcrafted evaluation functions refined by human experts over several decades. Shogi programs were built the same way, around a highly optimised alpha-beta search.

AlphaZero replaced all of that with a deep neural network, which takes a position and outputs move probabilities and an evaluation, plus general-purpose Monte Carlo tree search. The network starts from random weights and learns only from the results of games against itself. Where AlphaGo Zero assumed a binary win or loss, AlphaZero optimised the expected outcome, draws included, because chess and shogi can both end in a draw.

The search was strikingly different too. According to the preprint, AlphaZero examined just 80,000 positions per second in chess and 40,000 in shogi, against 70 million for Stockfish and 35 million for Elmo. The paper describes its network-guided focus on the most promising variations as arguably a more "human-like" approach to search, as originally proposed by Claude Shannon.

The December 2017 preprint

Each game was trained separately for 700,000 steps (mini-batches of 4,096 positions), using 5,000 first-generation TPUs to generate self-play games and 64 second-generation TPUs to train the networks. According to the preprint, AlphaZero outperformed Stockfish after 4 hours of training, Elmo after less than 2 hours, and the version of AlphaGo that played Lee Sedol after 8 hours.

In these matches AlphaZero and AlphaGo Zero ran on a single machine with 4 TPUs, while Stockfish and Elmo played at their strongest skill level using 64 threads and a 1 GB hash.

The dispute over conditions

The results made headlines, but the conditions drew criticism. On 8 December 2017 Chess.com published in full a comment from Tord Romstad, one of Stockfish's original authors. "The match results by themselves are not particularly meaningful because of the rather strange choice of time controls and Stockfish parameter settings," he wrote. At a fixed minute per move, Stockfish could make no use of its time-management heuristics for spending extra time at critical moments. The version used was a year old, ran with far more search threads than had ever been tested much, and had hash tables far too small for that many threads. He believed the share of draws would have been much higher under more normal conditions.

Romstad also granted that AlphaZero could have played better with more work, while calling the "4 hours of learning" highly misleading given the massive hardware used during those hours. Comparing a conventional program on ordinary computers with fundamentally different techniques on custom hardware not available for purchase was, he said, "very much a comparison of apples to orangutans."

The 2018 Science paper

The full evaluation a year later changed the conditions. According to DeepMind's summary, every game was played at three hours per game plus 15 seconds per move. Stockfish and Elmo used 44 CPU cores, as in the TCEC world championship; AlphaZero and AlphaGo Zero used a single machine with 4 first-generation TPUs and 44 CPU cores.

DeepMind says AlphaZero also won matches that started from common human openings, from the 2016 TCEC opening positions, against the latest development version of Stockfish and against a Stockfish variant with a strong opening book. Training took roughly 9 hours for chess, 12 hours for shogi and 13 days for Go. Some figures differ between the 2017 preprint and the 2018 summary, such as the time to pass Elmo (2 hours) and the time to pass the version of AlphaGo that played Lee Sedol (30 hours).

What it meant for players and programmers

DeepMind's summary quotes the shogi champion Yoshiharu Habu. AlphaZero, he observed, makes moves that go against shogi theory and look perilous to a human, such as moving the king to the centre of the board, yet remains in control. "Its unique playing style shows us that there are new possibilities for the game." Former world chess champion Garry Kasparov said that self-taught expert machines not only perform incredibly well, but "we can actually learn from the new knowledge they produce." Grandmaster Matthew Sadler and Woman International Master Natasha Regan analysed thousands of its games for their book Game Changer (New in Chess, January 2019).

Romstad pointed to another effect. For chess-engine programmers, and for programmers in many other domains, he wrote, the rise of machine-learning techniques that need massive hardware to be effective was "a little disheartening." Even if AlphaZero-like programs could one day run on ordinary computers, the resources needed to create them would be beyond hobbyists and average-sized companies, and the days of hundreds of unique engines, each with its own quirks, would be gone. The pattern that later defined generative AI, with the strongest systems built only by organisations that command enormous compute, was put into words here early.

DeepMind's own history is on the DeepMind and Demis Hassabis pages, the machine that beat Kasparov in 1997 on the Deep Blue page, and the wider story on the AI timeline.