Claude Shannon — Founder of information theory

Claude Elwood Shannon (30 April 1916 — 24 February 2001) erected, in a single paper, the mathematical framework that treats information as a measurable physical quantity. Digital communication, data compression, cryptography, and modern machine learning all stand on his foundation: every technology that passes through the unit known as the "bit" inherits from Shannon.

Background

Shannon was born in Gaylord, Michigan. His father was a local judge; his mother was a high-school principal. As a child he built radios and electrical devices, and in one anecdote he later told he ran a telegraph line to a friend's house to send messages by code.

He took a double bachelor's degree in electrical engineering and mathematics at the University of Michigan (1936) and continued at MIT for his master's and doctorate, completing a PhD in mathematics in 1940 with a thesis on theoretical genetics. His 1937 master's thesis, A Symbolic Analysis of Relay and Switching Circuits, was the first systematic demonstration that George Boole's nineteenth-century algebra of logic mapped directly onto electrical switching circuits — the mathematical footing of digital circuit design. Howard Gardner later called it "possibly the most important, and also the most famous, master's thesis of the twentieth century".

Shannon joined Bell Labs in 1941, became a full professor at MIT in 1956, and attended the Dartmouth workshop the same year. He retired from MIT in 1978 and spent his later years in Winchester, Massachusetts; he died in 2001, aged 84, of complications from Alzheimer's disease.

Major contributions

1937: Boolean algebra and relay circuits

Shannon's master's thesis showed that the behaviour of any electrical switching circuit could be described entirely in two-valued logic — Boolean algebra. Switching circuit design had until then relied on craft heuristics; with Shannon's theorem the circuit became an object that could be written as an equation, simplified, and verified. Transistors, integrated circuits, and microprocessors all start here.

1948: information theory

A Mathematical Theory of Communication, published in the Bell System Technical Journal in July and October 1948, ranks among the most important papers in the history of computer science. In it Shannon introduced:

• The bit (binary digit), the unit of information. The word "bit" was attributed in a footnote to John Tukey.
• Entropy H = -Σ p(x) log p(x), a measure of the uncertainty of a source. The formal analogy with thermodynamic entropy was, by Shannon's own account, suggested to him by John von Neumann.
• Channel capacity, the maximum information rate transmissible through a noisy channel.
• The noisy-channel coding theorem, an existence proof that any rate below capacity admits a coding scheme of arbitrarily small error.

With this one paper, communications engineering ceased to be a craft and became a mathematical optimisation problem. Wi-Fi, 5G, satellite links, optical fibre, JPEG, MP3, H.265, and quantum information theory all sit within its reach.

1949: cryptography

A previously classified Bell Labs internal report, Communication Theory of Secrecy Systems, was published in 1949. In it Shannon proved the conditions for information-theoretic perfect secrecy: that the key must be at least as long as the plaintext, truly random, and used only once. This is the mathematical justification of Vernam's one-time pad and the starting point of modern cryptography.

1950: chess and the computer

Programming a Computer for Playing Chess (1950) was the first theoretical paper on computer chess. The minimax search, evaluation function, and quiescence search outlined there ran directly into Deep Blue (1997) and, more distantly, AlphaGo (2016).

Juggler, unicyclist, Theseus

Shannon was as well known for play as for mathematics. He rode a unicycle through Bell Labs corridors while juggling, and wrote a paper on the theory of juggling (the minimum-throw theorem). His "Ultimate Machine" — a box whose only function was that, when switched on, an arm emerged from inside to switch itself off — is preserved at MIT as a joke about the self-reference of computing.

In 1961 he and Edward O. Thorp built a small concealed device for predicting roulette outcomes from physical observation. It is widely cited as the first wearable computer.

Awards and honours

• 1940 Alfred Noble Prize (American Society of Civil Engineers, awarded for the master's thesis)
• 1966 National Medal of Science (United States)
• 1972 Harvey Prize
• 1985 Kyoto Prize in Basic Sciences

Shannon was never awarded the ACM Turing Award; the conventional explanation is that information theory was regarded as broader than "computer science" and outside the prize's scope.

Legacy

The bits Shannon defined now flow at the scale of zettabytes per second across the planet. Language models from the Transformer (2017) through to GPT-4 (2023) operate on real-valued vectors, but the operation underneath training and inference is Shannonian — minimising the entropy of a predicted next-token distribution.

Information theory is the source of computer science's theoretical dignity vis-à-vis physics. Placing "information" as a third fundamental quantity alongside matter and energy was a philosophical move that reshaped the terrain of science after Shannon.