LeNet — The Convolutional Network That Read Zip Codes and Checks

LeNet is the name of a line of convolutional neural networks (CNNs) developed by Yann LeCun and colleagues at AT&T Bell Laboratories. The starting point is "Backpropagation Applied to Handwritten Zip Code Recognition", published in Neural Computation in December 1989 (vol. 1, no. 4, pp. 541–551) by seven authors including LeCun, Bernhard Boser and John S. Denker. It trained a network with backpropagation to read handwritten zip-code digits from US mail directly from the image. The best-known version, LeNet-5, was described in detail in a November 1998 paper in the Proceedings of the IEEE, and according to that paper a check-reading system built on this line of work was used by several US banks. In 2019 ACM cited LeCun, in its Turing Award announcement, as the first to train a CNN system on images of handwritten digits.

The 1989 paper — reading zip codes

The 1989 abstract states that the ability of learning networks to generalise can be greatly enhanced by providing constraints from the task domain, and that the paper shows how such constraints can be built into the architecture of a backpropagation network. The task was handwritten zip-code digits provided by the US Postal Service, and a single network learned the entire recognition operation, from the normalised image of a character to the final classification.

The key ideas were weight sharing and feature maps. So that the same feature — a line ending, a corner — could be found wherever it appeared in the image, small detectors with identical weights were laid across the whole image. This cut the number of free parameters sharply while keeping many connections. The paper reports that fully connected, unconstrained networks it tried generalised quite badly, and notes that some of the learned detectors looked remarkably like feature detectors found in biological vision systems.

Finding the zip code on the envelope and cutting it into single digits was done by Postal Service contractors. The paper also reports an implementation with an off-the-shelf DSP board attached to a PC, which recognised digits captured by a camera at 10 to 12 characters per second.

LeNet-5 and MNIST (1998)

In 1998 LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner published "Gradient-Based Learning Applied to Document Recognition" in the Proceedings of the IEEE (vol. 86, no. 11, pp. 2278–2324). The LeNet-5 described there has seven layers, not counting the input, and according to the paper 345,308 connections but only 60,000 trainable parameters, thanks to weight sharing.

How the MNIST (Modified NIST) dataset built for this paper came about is itself a lesson in how training data shapes results. According to the paper, Special Database 3, which NIST had designated for training, was collected among Census Bureau employees, while Special Database 1, designated for testing, was collected among high-school students, and the former was much cleaner and easier to recognise. So that conclusions would not depend on which set was used for what, LeCun and colleagues remixed the two. The question of whose writing a machine learns from, and how that decides what it does well or badly, would later become a public issue in face recognition and language models.

Reading bank checks

The 1998 paper also describes a system for reading the amount on bank checks. The abstract says it is deployed commercially and reads several million checks per day. Another passage says the system was integrated into NCR's line of check-recognition systems for the banking industry and had been fielded in several US banks since June 1996. Yet the paper's introduction speaks of millions of checks per month, so the scale figures are not consistent even within the paper. All of them are the developers' own descriptions.

According to the paper, an independent test by systems integrators in 1995 found the system superior to other commercial courtesy-amount reading systems. In other words, LeNet did not suddenly replace purely manual work; it entered banking by improving the accuracy of automatic reading that already existed. The paper gives no figure for how much manual work was reduced. ACM's 2019 Turing Award announcement also credits Bengio with combining neural networks with hidden Markov models, ideas that were incorporated into a system used by AT&T and NCR to read handwritten checks.

What came after

According to ACM's announcement, convolutional neural networks are today an industry standard in computer vision and are also used in speech recognition, speech synthesis, image synthesis and natural language processing, with applications from autonomous driving and medical image analysis to voice assistants and information filtering. The 1998 paper's conclusion observed that increasing the role of learning had invariably improved recognition systems, and that convolutional networks had eliminated the need for hand-crafted feature extractors.

The road from LeNet to the deep-learning era was not a straight line, though. What made CNNs widely recognised as the mainstream of image recognition was AlexNet in 2012, trained on GPUs with large amounts of data, which in turn rested on the construction of ImageNet. LeNet's structure of convolution and weight sharing carried straight into it. On the risks of AI, LeCun has taken a different position from Hinton, with whom he shared the Turing Award (see Hinton's departure from Google).