word2vec — Words as Vectors That Carry Meaning

word2vec is both a method for turning words into real-valued vectors of a few hundred dimensions and the name of its implementation. It began with "Efficient Estimation of Word Representations in Vector Space", a paper that Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean of Google posted to arXiv on 16 January 2013. It set out two models that learn, quickly and from large amounts of text, vectors in which words with similar meaning or grammatical role lie close together. In August that year Google open-sourced the code as word2vec, and in October an improved follow-up paper was posted and then accepted at NIPS 2013.

Representing words as vectors was not new in itself. What was new is stated plainly in the abstract: high-quality word vectors could be learned from a 1.6-billion-word data set in less than a day. The idea of the vectors themselves is covered in the embedding glossary entry; this page follows what happened in 2013 and what it left behind.

Two shallow models: CBOW and Skip-gram

Earlier neural language models were heavy structures with hidden layers, and the paper notes that, as far as the authors knew, none had been trained successfully on more than a few hundred million words. Mikolov and colleagues observed that most of the cost came from the non-linear hidden layer, removed it, and proposed two simple architectures.

The figures are from Table 3 of the paper, with all models trained on the same data. CBOW was slightly better on syntactic relationships, and Skip-gram much better on semantic ones. For the large-scale runs the team used a Google News corpus of about 6 billion tokens, with the vocabulary limited to the 1 million most frequent words.

"King − Man + Woman" and the test set

The example most people associate with word2vec is that taking King, subtracting Man and adding Woman gives Queen. The paper puts it this way: vector("King") − vector("Man") + vector("Woman") results in a vector that is closest to the vector representation of the word Queen. But the paper cites this as a result shown in earlier work by Mikolov and colleagues, not as its own discovery.

What the 2013 paper added was a systematic way to measure such relationships. It collected questions of the form "which word is similar to small in the same sense as biggest is similar to big?" into a test set of 8,869 semantic and 10,675 syntactic questions. Answers are found by adding and subtracting vectors and taking the nearest word by cosine distance. The paper says that well-trained vectors can capture subtle semantic relationships, such as France being to Paris as Germany is to Berlin.

Release and follow-up: August and October 2013

According to the revised paper, after the first version was written the team published single-machine, multi-threaded C++ code for both CBOW and Skip-gram, along with more than 1.4 million vectors for named entities trained on more than 100 billion words. On 14 August 2013 the Google Open Source Blog, in a post by Mikolov, Ilya Sutskever and Quoc Le, announced word2vec as open source. The post showed the model learning the relationship between countries and their capitals from news articles without human supervision, and listed knowledge extraction, machine translation, question answering and conversational systems as possible uses.

Ilya Sutskever joined the authors of the follow-up, "Distributed Representations of Words and Phrases and their Compositionality". It introduced subsampling of frequent words and negative sampling to lighten the computation, and said an optimised single-machine implementation could train on more than 100 billion words in a day. It also tackled the problem that adding the meanings of "Air" and "Canada" does not give the airline Air Canada, and showed how to learn one vector for a whole phrase. In 2023 NeurIPS gave the paper its Test of Time Award, noting more than 40,000 citations.

What changed: a part for everything downstream

The first paper ended by predicting that high-quality word vectors would become an important building block for future language applications. The idea of replacing words with vectors that preserve closeness of meaning did become the entry point of later language models: the 2017 Transformer also converts its input tokens to vectors with learned embeddings before processing them. Yet the follow-up paper itself listed the limits: word representations ignore word order and cannot represent idiomatic phrases. Representations that read a whole context were left to the next generation, such as BERT.

For users, the change spread quietly. A 2016 paper by Bolukbasi and colleagues described word embeddings being used in downstream applications such as document ranking, sentiment analysis and question retrieval, and counted hundreds or thousands of papers on uses from web search to parsing CVs. Because the code and vectors were public, researchers and companies without large computing budgets could use the same part.

The bias finding: 2016

The cost of that convenience was soon measured. In a July 2016 paper, Tolga Bolukbasi and colleagues at Boston University and Microsoft Research reported that even embeddings trained on Google News articles exhibit gender stereotypes "to a disturbing extent". As the paper's title has it, man is to computer programmer as woman is to homemaker.

The authors argued that the wider embeddings are used, the more they risk amplifying such biases. In settings such as CV parsing or search, bias in the training text can bear directly on someone's opportunities. The paper proposed a method to reduce the bias, but it also set out the opposing views: that embedding bias merely reflects bias in society, which is what should be fixed, and that in some applications the biased statistics could be useful. The authors nevertheless recommended erring on the side of neutrality, given the risk of amplifying stereotypes.

For the wider story, see the AI timeline and the Google page.