"Attention Is All You Need" — The Transformer Paper

The first footnote of "Attention Is All You Need" is unusual for a paper with eight names on it. It begins: "Equal contribution. Listing order is random." Then it says who did what. Jakob Uszkoreit "proposed replacing RNNs with self-attention and started the effort to evaluate this idea." Ashish Vaswani, with Illia Polosukhin, "designed and implemented the first Transformer models". Noam Shazeer "proposed scaled dot-product attention, multi-head attention and the parameter-free position representation". Niki Parmar designed, tuned and evaluated model variants; Llion Jones was responsible for the initial codebase, efficient inference and visualisations; Łukasz Kaiser and Aidan Gomez built tensor2tensor, the library that replaced the earlier codebase.

That footnote is worth reading before the architecture, because it explains why the paper is credited to a group rather than to one inventor. The name that appears first is there by chance.

The Problem the Transformer Paper Set Out to Solve

By 2017, neural machine translation was dominated by encoder-decoder models built from recurrent networks — plain RNNs, LSTMs and gated recurrent units. The paper's introduction states the limitation directly. A recurrent model computes a hidden state for each position from the hidden state of the position before it, and "this inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths".

Attention itself was not new. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio had added a soft-alignment mechanism to a recurrent translator in a paper first posted in September 2014, letting the decoder look back over the whole source sentence instead of squeezing it into one fixed-length vector. But as the 2017 paper notes, "in all but a few cases" such attention was "used in conjunction with a recurrent network." Other groups were attacking the sequential bottleneck with convolutions — the paper names the Extended Neural GPU, ByteNet and ConvS2S.

The Transformer's proposal was the radical version: keep the attention, and throw away both the recurrence and the convolution.

How the Transformer Is Built

The model in the paper is still an encoder-decoder, the shape translation systems already had. What changed is what fills each half.

• Self-attention. Every position in a sentence computes a weighted combination of every other position. The weights come from a dot product between a "query" for one position and a "key" for each of the others, scaled by the square root of the key dimension — hence scaled dot-product attention.
• Multi-head attention. Instead of one attention calculation, the base model runs eight in parallel on smaller projections and concatenates the results, so different heads can track different relationships.
• Positional encoding. With no recurrence, the model has no built-in sense of word order. The paper adds fixed sine and cosine signals of different frequencies to the input embeddings; it reports that learned position embeddings gave nearly identical results.
• Stacks. The base model uses six identical layers in the encoder and six in the decoder, a model width of 512, and about 65 million parameters. The "big" variant is about 213 million.

The Results Reported in the Paper

The claims were about translation quality and about cost.

The paper also applied the model to English constituency parsing, a task with no translation in it, to argue that the architecture generalised. The discrepancy in the French figure is in the paper itself; this page reports both rather than choosing one.

The cost line was the part with consequences. The authors wrote that the Transformer "can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs." Training that parallelises across a whole sentence is training that scales with more hardware, and that property — more than the BLEU scores — is what later work exploited.

Who Wrote It, and Where

The work was done at Google. Each author carries one of two footnotes: "Work performed while at Google Brain" or "Work performed while at Google Research." The affiliations printed under the names are slightly messier than that summary. Aidan N. Gomez is listed at the University of Toronto, and Illia Polosukhin with a personal email address rather than a Google one.

The paper's history on arXiv is also more than one date. Version 1 was posted on 12 June 2017; three more revisions followed by the end of that month; version 5 appeared on 6 December 2017, and the paper was presented at the 31st Conference on Neural Information Processing Systems (NIPS 2017) in Long Beach, California. Two further versions were added in July and August 2023.

What Was Built on the Transformer

The conclusion of the paper was modest about scope. It planned "to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video."

What happened first was in language. OpenAI's paper by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, "Improving Language Understanding by Generative Pre-Training" — the model now called GPT — states plainly: "For our model architecture, we use the Transformer". It used a 12-layer decoder-only version, pre-trained on unlabelled text and then fine-tuned. Google's BERT, first posted on 11 October 2018, put the architecture in its name: Bidirectional Encoder Representations from Transformers. The two took opposite halves of the original model — GPT the decoder, BERT the encoder — and between them set the template for the large language models that followed, up to and including ChatGPT.

The citation record reflects that. In 2025 Nature ranked the most-cited papers of the twenty-first century across five major databases, and, as the University of Toronto reported, "Attention is all you need" came seventh.

Placing the Paper

The Transformer did not introduce attention, and it did not by itself produce large language models; both claims are common and both overstate it. What the paper showed was narrower and more useful: that attention alone was enough to build a competitive sequence model, and that such a model could be trained on eight GPUs in half a day with the hardware of 2017. The attention mechanism came from the recurrent era; the scale came later, from the people who noticed what parallel training made possible.

For the earlier half of the story, see the neural-network revival of 2012; for where the Transformer sits among seventy years of AI, see the AI timeline.