IBM's Statistical Machine Translation — Turning Translation from Rules into Data

The paper usually taken as the starting point of statistical machine translation is "A Statistical Approach to Machine Translation", published in Computational Linguistics in June 1990 (vol. 16, no. 2, pp. 79–85) by Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John D. Lafferty, Robert L. Mercer and Paul S. Roossin of IBM's Thomas J. Watson Research Center. Instead of having linguists write rules of grammar and dictionaries, the team let a computer learn word correspondences and word-order probabilities from the English–French proceedings of the Canadian Parliament, and used them to translate French into English. The results were still rough, but the idea of turning translation from a problem of rules into a problem of data runs through Google Translate in 2006 to today's neural machine translation and large language models.

A 1949 idea, forty years on

As the paper's opening explains, the first person to suggest attacking machine translation with statistics and information theory was Warren Weaver in 1949, while he, Claude Shannon and others were developing that theory. Researchers soon abandoned the approach, raising many theoretical objections, but Brown and colleagues believed the real obstacles had been the weakness of the available computers and the shortage of machine-readable text to gather statistics from. By 1990 computers were five orders of magnitude faster than in 1950 and had hundreds of millions of bytes of storage, large corpora were readily available, and statistical methods had proven their value in speech recognition. "We feel that it is time to give them a chance in machine translation," the paper says. The National Research Council's history notes that Jelinek, one of the authors, and IBM's speech research group contributed enormously to speech-recognition technology through statistical language modelling.

The same authors had presented a preliminary version, "A Statistical Approach to Language Translation", at the COLING conference in 1988.

Translation as probability

The framework is simple. Given a French sentence T, the chance of error is smallest if one picks the most probable English sentence S that could have produced it. By Bayes' theorem this means finding the S that maximises the product of a "language model" Pr(S) and a "translation model" Pr(T|S). The language model takes care of whether the English is natural; the translation model takes care of which words correspond. In the paper's words, the translation model suggests which source words might have produced the observed words, and the language model suggests an order in which to place them. Nobody told the system which words corresponded to which (the alignment); it estimated them from the paired sentences with the EM algorithm, as speech recognition did.

Canada's parliamentary record as a textbook

The training material was Hansard, the record of the Canadian Parliament's proceedings, kept in both English and French. According to the paper, IBM obtained about 100 million words of English and the corresponding French from the Canadian government, and extracted about three million sentence pairs with a statistical algorithm based on sentence length. Roughly 99% of the pairs were genuine translations of one another.

Canada's practice of keeping official records in two languages had, without intending to, produced a large parallel corpus, and that corpus became a textbook for machine translation. The pattern of public records and human writing being used for machine learning in ways their authors never anticipated appears here early.

The results — 48% and 60%

In the French-to-English experiment, 73 translated sentences were sorted into five categories.

The authors counted the first three categories as reasonable translations and, on that criterion, reported success 48% of the time. One of them also counted the keystrokes needed to correct the flawed output: 776, against 1,916 to type the Hansard translations from scratch, which they took as a reduction in work of about 60%. The paper itself adds the condition "to the extent that translation time can be equated with key strokes". And the translation model had used only about 1% of the available data, the language model about 10%.

Measuring how much post-editing effort a system saves is an early form of the question that would later surround machine translation: does it replace human translators, or does it become a tool that drafts for them?

From rules to data

When the 1990 paper appeared, working machine translation was rule-based. In a blog post on 28 April 2006, Google researcher Franz Och explained that most commercial systems of the day relied heavily on linguists defining vocabularies and grammars, whereas Google's approach fed the computer billions of words of text and built a translation model with statistical learning. Google launched statistical machine translation between Arabic and English at that time.

Ten years later, in 2016, Google moved to neural machine translation, and the Transformer of the following year was also introduced as a machine-translation paper. The 1990 idea — large amounts of parallel text rather than rules written by linguists — also connects to questions that now run through all of AI: how translation work is done, and who supplies the training data and who profits from it.