AlexNet — The Deep-Learning Era Begins

Submissions to the ImageNet Large Scale Visual Recognition Challenge 2012 closed at 23:00 GMT on 30 September 2012. The results were presented a fortnight later, at the challenge workshop held with ECCV in Florence on 12 October, and the full standings went out on 13 October.

The classification leaderboard was headed by the University of Toronto's entry, 'SuperVision': 15.3% top-5 error (0.15315) for the run that used the ImageNet Fall 2011 release as extra training data, and 16.4% (0.16422) for the run on supplied data only. The best result from any other team was 26.2% (0.26172), from the University of Tokyo's ISI group.

A ten-point gap on an image-recognition benchmark was, historically, impossible. XRCE had won the 2011 challenge at 25.8% (0.25770); two graduate students and their advisor cut the error nearly in half in a single year.

And they had done it with neural networks — a technique that had been out of the computer-vision mainstream for years.

A Team of Three

• Alex Krizhevsky — then a PhD student. He wrote the architecture and the CUDA kernels that made it run on the GPUs.
• Ilya Sutskever — also a PhD student. Later co-founder and Chief Scientist of OpenAI.
• Geoffrey Hinton — their advisor. Working on neural networks since the 1970s, and one of the very few who had stayed on that side through the second AI winter.

By CHM's account of the project's origin, it was Sutskever who in 2011 persuaded Krizhevsky to train a network on ImageNet — Sutskever believed network performance would scale with the amount of data available, and Fei-Fei Li's ImageNet, completed in 2009, was orders of magnitude larger than any image dataset before it. Hinton's own one-sentence summary, given to CHM years afterwards: "Ilya thought we should do it, Alex made it work, and I got the Nobel Prize."

The paper was titled "ImageNet Classification with Deep Convolutional Neural Networks", and the competition entry was filed under the team name 'SuperVision'. 'AlexNet', after the first author, is a nickname the community attached afterwards.

What the Network Was

Eight learned layers: five convolutional, three fully-connected, ending in a 1000-way softmax. About 60 million parameters and 650,000 neurons. Input was a 224×224×3 crop taken from images downsampled to 256×256; apart from subtracting the mean pixel value over the training set, the paper reports no preprocessing at all — the network was trained on "the (centered) raw RGB values of the pixels".

Before AlexNet, image recognition was the art of feature engineering. SIFT, HOG, SURF — algorithms that extracted corners, gradients and textures, designed by hand and fed to a classifier such as an SVM. The longest part of writing a paper was usually the tinkering with features. AlexNet's alternative was to let the network learn the features itself, end to end, on the 1.2 million labelled training images of ILSVRC.

The paper ranks its own novelties by importance, and the ranking is instructive:

1. ReLU. Instead of the saturating tanh(x) or sigmoid, neurons used f(x) = max(0, x) — Rectified Linear Units, following Nair and Hinton. Figure 1 shows a four-layer CNN with ReLUs reaching 25% training error on CIFAR-10 six times faster than the same network with tanh units. The authors state flatly that without it, "we would not have been able to experiment with such large neural networks for this work."

2. Two GPUs. Covered below; it is the second most important item in the paper's own ordering.

3. Local response normalisation, a lateral-inhibition scheme across adjacent kernel maps.

4. Dropout. Each hidden neuron in the two fully-connected layers had its output set to zero with probability 0.5 during training, so every input effectively sampled a different architecture from a pool sharing one set of weights; at test time all neurons ran with their outputs multiplied by 0.5. This costs roughly a factor of two in training time and buys the ensemble effect that a network with 60 million parameters badly needs.

Augmentation added translations, reflections and a PCA-based jitter of RGB intensities — the latter alone reducing top-1 error "by over 1%". And depth was load-bearing rather than decorative: removing any single convolutional layer, "each of which contains no more than 1% of the model's parameters", made the results worse.

Two GPUs, and Why That Was the Constraint

The GPU is usually described as the thing that made AlexNet fast. In the paper it is the thing that made AlexNet shaped the way it is.

"A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it." That ceiling, not a design preference, is why the network was split across two cards. The scheme puts half the kernels on each GPU and lets them communicate only at certain layers — layer 3 reads from all of layer 2, but layer 4 reads only from the layer-3 maps on its own card — so the amount of cross-GPU traffic could be tuned against the amount of computation. Measured against a one-GPU net with half as many kernels per convolutional layer, the split reduced top-1 and top-5 error by 1.7 and 1.2 points, and trained slightly faster.

Training ran for roughly 90 cycles through the 1.2 million images and took five to six days on two NVIDIA GTX 580 3GB cards — consumer gaming hardware. The learning rate started at 0.01 and was divided by ten whenever validation error stopped improving, three times before the run ended. The CUDA convolution code was released publicly with the paper.

What it displaced is visible in the paper's own comparison table for ILSVRC-2010, the year for which test labels were public:

As the first showing in which GPU-trained deep learning took a large-scale benchmark, AlexNet also reshaped NVIDIA. From 2012 onward the company shifted from making gaming GPUs to being the central infrastructure supplier of AI computation. On 18 June 2024 its market capitalisation briefly made it the most valuable company in the world.

The Citation Trajectory

Jeff Dean, announcing the source-code release in 2025, called it "one of the most cited papers of all time" — and the counters bear him out. Semantic Scholar's record for the paper stood at 130,683 citations on 9 August 2026, with 13,428 of them classified as influential. For a conference paper whose contribution was, in its authors' own framing, an assembly of existing ideas run at a new scale, that is an unusual afterlife.

The Code, Thirteen Years Later

On 20 March 2025, the Computer History Museum, in partnership with Google, released and preserved the AlexNet source code — the code Krizhevsky wrote and refined over the year following Sutskever's suggestion, on a machine with two NVIDIA cards. A Google team led by David Bieber worked with CHM for five years to get it cleared for public release.

"Before AlexNet, very few machine learning researchers used neural networks," CHM's release notes. "After it, almost all of them would."

The Third Summer Opens

AI's history has had two 'winters', framing two 'summers' — the first from 1956 to the mid-1970s, the second the expert-systems boom of the 1980s.

What AlexNet began was the third summer — the deep-learning era. Unlike the previous summers, more than a decade in, by 2026, it is still going.