Deep Belief Nets — In 2006, Deep Neural Networks Learned Again

A deep belief net (DBN) is a generative model with many hidden layers, proposed by Geoffrey Hinton, Simon Osindero and Yee-Whye Teh in "A fast learning algorithm for deep belief nets", published in Neural Computation (volume 18, issue 7) in July 2006. Using a fast greedy method that trains restricted Boltzmann machines (RBMs) one layer at a time and stacks them, the paper showed that deep networks, which had been very hard to train, could be trained after all. On 28 July of the same month Hinton and Ruslan Salakhutdinov published a paper in Science applying the same pre-training to deep autoencoders. When 2006 is called the year deep learning restarted, it is mainly these two papers that are meant.

Background: neural networks out of favour

In their 2015 review in Nature, Yann LeCun, Yoshua Bengio and Hinton described the late 1990s this way: "neural nets and backpropagation were largely forsaken by the machine-learning community and ignored by the computer-vision and speech-recognition communities." It was widely thought, they wrote, that learning multistage feature extractors with little prior knowledge was infeasible, and that gradient descent would get trapped in poor local minima.

Backpropagation itself had been popularised by Rumelhart, Hinton and Williams in 1986 (backpropagation). The difficulty was that deeper networks did not train well.

The two papers of July 2006

In the Neural Computation model, the top two layers form an undirected associative memory and the layers below have directed connections that generate the pixels of an image. Each layer is first trained as an RBM, from the bottom up, with each higher layer learning from the output of the one below. The paper shows that each added layer improves a variational bound on the whole generative model. The network is then fine-tuned with what the authors call the up-down algorithm, a contrastive version of the wake-sleep algorithm.

On the MNIST handwritten digits (60,000 training and 10,000 test images), a network with three hidden layers and about 1.7 million weights reached a 1.25% error rate on the official test set. The paper compares this with 1.5% for the best backpropagation networks that are not hand-crafted for the task, and 1.4% for support vector machines. Training ran in Matlab on a 3 GHz Xeon: a few hours per layer for the greedy stage and about a week in all.

The Science paper used the same pre-training to initialise deep autoencoders that compress a 784-pixel image into 30 numbers. Without pre-training, a very deep autoencoder always reconstructed the average of the training data, even after prolonged fine-tuning. With it, reconstructions were far better than PCA's, and on retrieval over more than 800,000 newswire stories the autoencoder outperformed latent semantic analysis. The paper closes: "It has been obvious since the 1980s that backpropagation through deep autoencoders would be very effective for nonlinear dimensionality reduction, provided that computers were fast enough, data sets were big enough, and the initial weights were close enough to a good solution. All three conditions are now satisfied."

Other papers in 2006 pursued the same idea, among them Bengio and colleagues on greedy layer-wise training and a sparse-representation paper from LeCun's group. The Nature review cites them together as the basis of the revival.

How the history is told: the 2006 story and its critics

The Nature review states: "Interest in deep feedforward networks was revived around 2006 (refs 31-34) by a group of researchers brought together by the Canadian Institute for Advanced Research (CIFAR)." The first major application, it says, was speech recognition: in 2009 a pre-trained deep net set records on a standard small-vocabulary benchmark, and by 2012 versions of it were being developed by many of the major speech groups and deployed in Android phones.

The strongest objection came from Jürgen Schmidhuber of IDSIA in Switzerland. In a critique published in June 2015 he wrote of that passage: "Here they refer exclusively to their own labs, which is misleading." He pointed out that by 2006 many researchers had used deep nets of the Ivakhnenko type for decades, and argued that his own unsupervised pre-training of deep recurrent networks with the "history compressor" in the early 1990s was very similar in spirit and justification to the post-2006 methods. The story that begins in 2006 is largely the account of the people who wrote those papers, and it is worth reading it that way.

What followed, and the impact

Layer-wise pre-training did not stay at the centre for long. The Nature review explains that rectified linear units (ReLUs) made it possible to train deep supervised networks without unsupervised pre-training, and that "Once deep learning had been rehabilitated, it turned out that the pre-training stage was only needed for small data sets." The 2012 breakthrough in image recognition, the review says, came from the efficient use of GPUs, ReLUs, dropout and data augmentation, and brought about a revolution in computer vision (AlexNet).

Even so, the 2006 papers brought researchers' attention and money back to deep networks. Better speech recognition reached smartphone users directly, and the lead in research passed from features designed by people to features learned from data. On 27 March 2019 ACM gave Bengio, Hinton and LeCun the 2018 Turing Award, headlining them as the fathers of the deep learning revolution; the announcement notes that Hinton was the founding director of CIFAR's Neural Computation and Adaptive Perception programme. Hinton shared the 2024 Nobel Prize in Physics (the AI Nobel prizes), and the year before he had left Google in order to speak about the risks of AI (Hinton leaves Google).

The person is covered on the Geoffrey Hinton page, the terms under deep learning and unsupervised learning, and the wider story in the AI timeline.