ImageNet — A Dictionary of Images Built by 50,000 People

ImageNet is a large-scale image database built at Princeton University by Jia Deng, Fei-Fei Li and colleagues. Their paper, "ImageNet: A Large-Scale Hierarchical Image Database", was presented at CVPR 2009, the computer-vision conference held in Miami in June 2009. At that point it held 3.2 million human-checked images for 5,247 concepts taken from WordNet, a lexical database of English. The checking was done on a large scale by workers on Amazon Mechanical Turk. The official site now lists 14,197,122 images in 21,841 concepts, free to researchers for non-commercial use.

In a sentence, ImageNet was an attempt to change image recognition through the scale of the data rather than through algorithms. The competition run on this data was the stage for deep learning's decisive win in image recognition in 2012. That win itself is covered on the AlexNet page; this page is about how the dataset was built and what it left behind.

From the idea to CVPR 2009

According to Dave Gershgorn's account in Quartz, Fei-Fei Li began thinking about the idea in 2006. That year she visited Princeton and met Christiane Fellbaum, a researcher on WordNet; months later she joined the Princeton faculty and started ImageNet in early 2007. In her 2015 TED talk Li said that "together with Professor Kai Li at Princeton University, we launched the ImageNet project in 2007."

Undergraduates labelling images proved too slow, and by Li's account the project failed to win any of the federal grants she applied for. The way out came from a chance hallway conversation with a graduate student who asked whether she had heard of Amazon Mechanical Turk, a service where people at computers around the world complete small online tasks for small payments. Even after that, the dataset took two and a half years to complete.

The paper set a goal of filling most of WordNet's roughly 80,000 noun concepts with 500 to 1,000 images each, on the order of 50 million images in all. It stressed the fineness of the hierarchy: no existing vision dataset, it said, offered images of 147 dog categories.

Gershgorn reports that CVPR gave the team only a poster rather than a talk, and that they handed out ImageNet-branded pens to drum up interest. The very idea that more data would produce better algorithms was still met with scepticism.

Sorted by 50,000 pairs of hands

ImageNet was built in two stages. First, candidate images for each concept were collected by querying several image search engines with the concept's synonyms, adding words from the parent concept and translating queries into Chinese, Spanish, Dutch and Italian. After duplicates were removed, the paper says, each concept had more than 10,000 candidates on average. Then Mechanical Turk workers looked at the candidates one at a time and voted on whether each fitted the concept.

Workers make mistakes and disagree. The paper had several workers judge each image independently and accepted an image only with a convincing majority. A "cat" needs few votes; a "Burmese cat" needs many. So for each concept at least ten workers first voted on an initial sample, and from that the system set how many agreeing votes were needed before an image counted as correct.

On scale, Li said in her 2015 talk that almost 50,000 workers from 167 countries helped to clean, sort and label nearly a billion candidate images, and that "At its peak, ImageNet was one of the biggest employers of the Amazon Mechanical Turk workers."

ILSVRC: from dataset to competition

From 2010 a part of ImageNet was used for an annual competition, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Its organisers write that it followed in the footsteps of Europe's PASCAL VOC challenge, established in 2005, and scaled it up by more than an order of magnitude: PASCAL VOC 2010 had 19,737 images in 20 classes, ILSVRC 2010 had 1,461,406 images in 1,000 classes. The classification task supplied about 1.2 million training images.

In Gershgorn's article, Alex Berg recalls that "The nice surprise was that people who trained their models on ImageNet could use them to jumpstart models for other recognition tasks."

The impact: whose photographs, and whose labour

Gershgorn's article presents ImageNet as a case in which data rather than algorithms changed the direction of AI research. ImageNet also left questions about whose labour built that data and whose faces are in it.

In September 2019 Kate Crawford and Trevor Paglen published "Excavating AI", a critique of ImageNet's person categories. The photographs had been taken from the internet without the knowledge or consent of the people in them. The labels, they wrote, included racial slurs and misogynistic terms, and the taxonomy treated appearance as a guide to character. They argued that training sets are not neutral raw material.

The maintainers responded. In a paper released in December 2019, Kaiyu Yang, Fei-Fei Li, Jia Deng, Olga Russakovsky and a colleague judged 1,593 of the 2,832 categories in the person subtree unsafe and planned to remove them, along with their 600,040 images. Of the remaining 1,239 categories, they found only 158 could be recognised visually at all. The paper acknowledges that WordNet's vocabulary was dated and that the filtering done during construction in 2009 had been imperfect. In March 2021 the team reported that blurring the faces of people who happen to appear in ILSVRC images reduced recognition accuracy by no more than 1.0%.

On labour, ImageNet set a precedent for building training data by paying workers around the world small sums for fragments of work. Gershgorn reports that start-ups soon began assembling their own datasets the same way, using Mechanical Turk workers.

The wider story is in the AI timeline, and questions of labour and ethics in the AI and society timeline. A case in which a home robot's training images leaked is on the Roomba page.