The Backpropagation Paper — Rumelhart, Hinton and Williams Show Hidden Units Can Learn

Backpropagation became widely known through a four-page paper by David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, "Learning representations by back-propagating errors", published in Nature on 9 October 1986 (vol. 323, pp. 533–536). In a multilayer neural network, the method traces the output error backwards from the output to the input and works out efficiently which way to move every connection weight. The paper showed experimentally that, trained this way, "hidden" units that are neither inputs nor outputs come to represent the features of the task on their own. The mathematics of computing derivatives in reverse, however, had been found in several fields before this paper. This page treats the 1986 paper not as the first invention but as the work that showed the method was effective for training multilayer networks and spread it; the grounds are the historical surveys by Griewank and Schmidhuber discussed below.

The mathematics of the method is explained in the glossary entry on backpropagation, and Hinton's career on his profile page. This page covers what the 1986 paper showed, how priority has been argued, and what followed for society.

What the paper showed

The abstract calls it a "new learning procedure" for networks of neurone-like units and explains it as follows. The procedure repeatedly adjusts the connection weights so as to minimise a measure of the difference between the network's actual output vector and the desired one. As a result, internal hidden units that belong to neither input nor output come to represent important features of the task domain. The abstract stresses that this ability to create useful new features is what distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure.

The perceptron, named in the abstract as the point of comparison, had only one layer of adjustable weights (see the perceptron). How to train multilayer networks with hidden layers was the problem left open, and the 1986 paper offered a practical answer. The same year, the collection Parallel Distributed Processing edited by Rumelhart and colleagues included a chapter by the same three authors, "Learning Internal Representations by Error Propagation". ACM cites that chapter among Hinton's achievements in announcing the 2018 Turing Award.

Who was first?

Computing derivatives in reverse had been discovered repeatedly in fields other than neural networks. In his 2012 paper "Who Invented the Reverse Mode of Differentiation?", the automatic-differentiation researcher Andreas Griewank writes that the reversal idea was suggested by several people from various fields since the late 1960s, if not earlier. The figure he highlights is Seppo Linnainmaa of Helsinki, who devised the method — what is now called reverse-mode automatic differentiation — in his 1970 master's thesis as a tool for estimating the effect of rounding errors on computed results. According to Griewank, Linnainmaa did not market it as a cheap way to compute gradients, so it found little resonance, and he published the results in English only in 1976.

For neural networks, the name that comes up is Paul Werbos. His 1974 doctoral thesis is often cited, but Jürgen Schmidhuber argues in an explainer on his website that Werbos applied efficient backpropagation to neural networks in 1982, and not yet in the 1974 thesis. Schmidhuber also criticises the 1986 paper: "This experimental analysis of backpropagation did not cite the origin of the method." This priority dispute is one in which Schmidhuber has long been a party himself.

The real dividing line is not the date of invention but who showed that it worked on multilayer networks and turned it into a tool researchers could pick up.

What changed

On 27 March 2019 ACM announced the 2018 Turing Award for Hinton, Yann LeCun and Yoshua Bengio, and placed backpropagation first among Hinton's achievements. The announcement says the work showed that backpropagation allowed neural networks to discover their own internal representations of data, making it possible to use them on problems previously thought to be beyond their reach, and that the backpropagation algorithm is standard in most neural networks today.

The spread can be traced through this site's timelines. In 1989 LeCun and colleagues at Bell Labs used backpropagation to read handwritten zip codes from US mail (LeNet). In 1992 TD-Gammon backpropagated reinforcement-learning errors the same way and taught itself backgammon through self-play. From AlexNet in 2012 to large language models such as GPT-3, this computation sits at the centre of training.

The effects outside the laboratory arrived only later. Two of the three authors were at an institute for cognitive science, and as the title says, the first interest was the cognitive-science question of learning representations. Decades on, the same computation drives products in image recognition, speech recognition, machine translation and generative AI, and who gains and who bears the costs — jobs displaced, training data and copyright, the electricity used for computation — has become a public dispute. That debate is gathered in the AI and society timeline.

Not how the brain does it — Hinton's own reservation

Backpropagation was not put forward as the brain's learning mechanism. In "The Forward-Forward Algorithm", a paper released in December 2022, Hinton wrote: "As a model of how cortex learns, backpropagation remains implausible". His reason is that there is no convincing evidence that the cortex explicitly propagates error derivatives backwards, and he has been testing a learning procedure that does without backpropagation.

The person who did most to popularise the most widely used learning method does not accept it as an account of the brain. The 1986 paper was a decisive engineering success, yet for its original motivation — understanding intelligence — it remains only part of the answer. How Hinton later came to speak publicly about the dangers of AI is told on the page about his departure from Google.