The Netflix Prize — $1 Million for Better Recommendations, and the Limits of Anonymity

The Netflix Prize was a public machine-learning competition launched by Netflix on 2 October 2006 to predict the ratings members would give to films. Netflix promised $1 million to the team that cut the prediction error of its Cinematch recommendation system by 10%, and on 21 September 2009 the prize went to a merged team of seven, BellKor's Pragmatic Chaos. The competition also became a defining case for data release and privacy: researchers showed that the "anonymised" behavioural data could be linked to individuals, a lawsuit followed, and in March 2010 Netflix dropped the second contest it had announced.

The rules: 100 million ratings and a 10% bar

According to the official rules, the training data consisted of more than 100 million ratings from over 480,000 randomly chosen, anonymous customers on nearly 18,000 titles. Accuracy was measured by root mean squared error (RMSE). Cinematch scored 0.9525 on the test subset, and the Grand Prize required 0.8572 or less, 90% of that figure. Entry was free, entrants did not have to be Netflix subscribers, and anyone could take part except residents of countries under US Treasury embargo. To win money, a team had to share its method with Netflix and describe it to the world.

In a 2012 post on its technology blog, Netflix explained that RMSE had been chosen as a proxy question, easier to evaluate and quantify than the real goal of better recommendations.

Three years of competition

In 2007 the first Progress Prize went to the Korbell team for an 8.43% improvement. Netflix's blog says the team reported more than 2,000 hours of work to arrive at a combination of 107 algorithms, and gave Netflix the source code. Netflix took the two best-performing pieces, matrix factorisation (which the community generally called SVD) and restricted Boltzmann machines (RBMs), adapted them to handle its own data of more than 5 billion ratings, and put them into production. The RBM is the same model that served as the building block of deep networks in 2006 (deep belief nets).

The 10% bar fell when rival teams joined forces. On the official leaderboard, BellKor's Pragmatic Chaos and The Ensemble, itself a combination of dozens of teams, both reached a test RMSE of 0.8567, a 10.06% improvement. BellKor's Pragmatic Chaos had submitted about 20 minutes earlier and won.

According to Fortune, the winning team's seven members were Bob Bell, Martin Chabbert, Michael Jahrer, Yehuda Koren, Martin Piotte, Andreas Töscher and Chris Volinsky, based in Austria, Canada, Israel and the United States; Volinsky and Bell were at AT&T Research. More than 40,000 teams from 186 countries had taken part. On the same day Netflix's chief executive Reed Hastings announced a second contest that would use implicit data such as viewing history and demographics to recommend films to members who do not rate them.

The winning ensemble itself never went into Netflix's product. The 2012 blog post says Netflix evaluated some of the new methods offline, "but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment." Netflix had launched streaming in 2007, a year after the prize began, and the blog explains that the problem changed when members were looking for something to watch right away. In the same post Netflix said that 75% of what people watch comes from some sort of recommendation.

Anonymous data that was not anonymous

About two weeks into the contest, on 18 October 2006, Arvind Narayanan and Vitaly Shmatikov of the University of Texas at Austin posted a paper on arXiv showing how to break the anonymity of the released data. A revised version was presented at the IEEE Symposium on Security and Privacy in May 2008.

The paper reports that "With 8 movie ratings (of which 2 may be completely wrong) and dates that may have a 14-day error, 99% of records be uniquely identified in the dataset." Two ratings with dates within three days were enough for 68%. The authors cross-matched records against public IMDb reviews, which people often post under their real names; out of respect for IMDb's terms of service they worked with only a few dozen IMDb users, and identified the Netflix records of two of them. For one, ratings made only on Netflix allowed strong guesses about his political orientation and religious views. The paper argued that even if most subscribers did not care, the existence of subscribers whose privacy could be compromised made this a serious privacy issue.

On 17 December 2009 four Netflix members filed a class action in the federal court for the Northern District of California. According to Netflix's annual report, they alleged that releasing data for the Netflix Prize breached subscribers' privacy and sought damages and an injunction under the Video Privacy Protection Act of 1988, which restricts disclosure of video rental records, and under California law. The lead plaintiff, Jane Doe, said the records could identify or permit inference of her sexual orientation and harm her ability to support her family.

On 12 March 2010 Netflix's chief product officer, Neil Hunt, wrote on the company blog that the Federal Trade Commission had asked how a sequel might affect members' privacy and that a lawsuit had been filed. "We have reached an understanding with the FTC and have settled the lawsuit with plaintiffs," he wrote, and Netflix had decided not to pursue the sequel. Hunt dated the sequel's announcement to 6 August 2009.

What it left behind

The gains were real. Netflix obtained core recommendation technology from outside researchers, and researchers got a large real-world dataset and a clear target. The finish, with rival teams merging to break the barrier, showed the strength of ensembles that combine many models.

For users the lesson was that anonymisation is not a complete defence. Removing names and IDs does not stop the pattern of a person's behaviour from pointing back to that person. Questions about behavioural data would return in a different form with Cambridge Analytica.

The wider story is in the AI timeline, and privacy and labour issues in the AI and society timeline.