IBM Watson Wins Jeopardy! — When Natural-Language Question Answering Beat Humans on TV

IBM's question-answering system Watson beat Ken Jennings and Brad Rutter, two of the most successful champions of the US quiz show Jeopardy!, in a contest broadcast in the US over three nights: 14, 15 and 16 February 2011. Two matches were spread across the three days, and Watson finished with $77,147 to Jennings's $24,000 and Rutter's $21,600. The games were taped in January 2011 at IBM's research centre in Yorktown Heights, New York.

Watson, a machine that parsed Jeopardy!'s quirky English clues and answered without any connection to the internet, gave IBM its biggest human-versus-machine spectacle since Deep Blue in 1997. The expectations the win raised, and the troubles of the medical business that followed, have also made it a standard example of the gap between AI marketing and AI capability.

Why Jeopardy!

According to IBM, the company had been looking for a new human-versus-machine challenge ever since Deep Blue beat Kasparov in 1997. In 2006 David Ferrucci, an IBM computer scientist with a background in AI, pitched his bosses on building a computer that could beat humans at Jeopardy!. The show reads out clues phrased as answers, contestants reply in the form of a question, and the clues are full of wordplay, irony and riddles.

"The goal is not to model the human brain," Ferrucci said, as IBM records it; the aim was a computer more effective at understanding and interacting in natural language, not necessarily in the way humans do it. He assembled a team of more than two dozen scientists, engineers and programmers at the Thomas J. Watson Research Center in Yorktown Heights, and the system took five years to perfect. It was named after IBM's first CEO, Thomas J. Watson Sr.

How Watson worked

As IBM explains it, a search engine returns a list of documents that may or may not be relevant, whereas Watson returned a direct answer. The more of its algorithms independently reached the same answer, the higher its confidence; below the threshold, it stayed silent. The design is set out in "Building Watson: An Overview of the DeepQA Project", published by Ferrucci and colleagues in AI Magazine in autumn 2010.

The contest, from the December 2010 announcement to the February broadcast

At the announcement IBM pledged to give all of its winnings to charity, and Jennings and Rutter half of theirs; according to PCWorld, IBM's share went to World Vision and World Community Grid. The opponents were formidable. Jennings had won 74 games in a row in the 2004–2005 season, earning more than $2.5 million; Rutter had won the highest cumulative amount in the show's history, $3,255,102.

Watson was not perfect. According to IBM, in the first match's Final Jeopardy!, in the category US Cities, it answered "What is Toronto?????" instead of "What is Chicago?", the five question marks signalling very low confidence. It still pulled away in the second match and won by a wide margin. Jennings added a line to his last written response: "I for one welcome our new computer overlords" (PCWorld).

What it meant for society

After the contest Jennings remarked, as IBM quotes him: "'Quiz show contestant' may be the first job made redundant by Watson, but I'm sure it won't be the last." The match put the idea of machines replacing knowledge work in front of a mass television audience.

IBM turned that momentum into a business. According to a 2019 IEEE Spectrum investigation, the day after the match IBM announced that Watson would be applied to medicine, promising its first commercial health-care offerings within 18 to 24 months; the projects announced that day did not yield commercial products. The article cites an oncology advisory tool built with MD Anderson Cancer Center, which MD Anderson cancelled in 2016 after spending $62 million on it. Robert Wachter of UCSF told Spectrum: "They came in with marketing first, product second, and got everybody excited." He also said that IBM, by being first to push AI into the clinic, had shown everyone else how hard the problems were.

On 21 January 2022 IBM announced an agreement to sell Watson Health's data and analytics assets, including MarketScan and Micromedex, to the investment firm Francisco Partners; financial terms were not disclosed. IBM kept the Watson name alive in watsonx, its generative-AI platform launched in 2023, as covered on the IBM page.

The Jeopardy! win showed statistical methods for handling the ambiguity of language reaching a level that could beat human champions on a television stage. Language AI then took a different route, through the Transformer of 2017 to large language models. The wider story is on the AI timeline.