BERT Released — Bidirectional Transformer Pretraining

On 11 October 2018, Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google AI posted "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" to arXiv (arXiv:1810.04805). The paper was formally presented at NAACL-HLT 2019 the following June, where it won Best Long Paper.

Its claim was new state of the art on eleven NLP tasks. The numbers in the abstract are all BERT-Large results.

The 93.2 on SQuAD v1.1 exceeded the human reference the leaderboard carried at the time (Test F1 91.2, as of 10 December 2018). That is not the same as a machine out-reading a person: SQuAD asks a model to select an answer span from a short supplied passage, and the human figure is annotator agreement on that narrow task.

Masked Tokens and Two Directions

BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only model derived from the 2017 Transformer paper (Vaswani et al.). It is specialised for turning input text into embeddings.

Two technical contributions stand out.

1. Bidirectional pretraining. GPT, which had appeared that June, was a left-to-right autoregressive model — processing any given position, it could not see context to the right. BERT instead used masked language modelling (MLM): 15% of the WordPiece tokens in each sequence are chosen for prediction, and of those, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged, with the model trained to recover the original. The asymmetry exists because [MASK] never appears at fine-tuning time; leaving some positions intact narrows the mismatch. The result is that every token attends to left and right context at once.

2. The "pretrain-then-fine-tune" paradigm. BERT was pretrained on BooksCorpus (800M words) and English Wikipedia (2,500M words), then adapted to downstream tasks — question answering, sentiment, classification — by adding one output layer and fine-tuning lightly. The era of designing a bespoke architecture per task ended.

Two sizes shipped: BERT-Base (L=12, H=768, A=12; 110M parameters) and BERT-Large (L=24, H=1024, A=16; 340M parameters). Base was deliberately sized to match OpenAI GPT for comparison. Large for its day — and three orders of magnitude smaller than GPT-3 (175B) two years later.

Open Weights and the BERT Family

One month after the paper, in November 2018, Google open-sourced the pretrained weights and TensorFlow code on GitHub. Having the weights let researchers and companies fine-tune on their own data, and the field moved almost overnight.

Derivatives poured out: RoBERTa (Facebook, July 2019, improved training recipe), ALBERT (Google, September 2019, parameter sharing), DistilBERT (Hugging Face, October 2019 — its authors reported it as 40% smaller and 60% faster while retaining 97% of BERT's language-understanding performance), XLNet, ELECTRA, plus multilingual mBERT, Japanese BERT, biomedical BioBERT, legal LegalBERT, and dozens more. "BERT-family" became standard vocabulary in NLP research.

Into Google Search

On 25 October 2019, Pandu Nayak — Google Fellow and VP of Search — announced BERT in production search. The scope was specific: "BERT will help Search better understand one in ten searches in the U.S. in English." Not all queries globally; one in ten US English searches. Google added that it was using a BERT model to improve featured snippets in the two dozen countries where that feature existed, with notable gains in Korean, Hindi and Portuguese.

Nayak called it "the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search" — Google's assessment of its own work.

On 9 December 2019 the rollout extended to more than 70 languages. At the Search On event on 15 October 2020, Prabhakar Raghavan said "BERT is now used in almost every query in English."

"Research to production in a year" was, for the time, extraordinary. It showed a direct pipeline from NLP academia to a service used by billions every day.

The End of Per-Task Architectures

Before BERT, NLP was task-specific: separate architectures for translation, question answering, sentiment, parsing. The dominant practice was bespoke design plus task-specific data.

BERT demonstrated a single large general-purpose model, pretrained once and fine-tuned cheaply for anything. That design philosophy is the foundation of every modern LLM — GPT-3, ChatGPT, Claude, Gemini — and it was established in 2018.

The contrast with GPT (OpenAI) is also instructive. GPT is decoder-only (strong at generation); BERT is encoder-only (strong at understanding). The field later passed through encoder-decoder designs (T5, BART) before settling on decoder-only plus scale as the dominant line. But across 2018–2022, BERT-family models were the protagonists of NLP.

119,000 Citations and a Surviving Niche

By Semantic Scholar's count, the BERT paper had been cited 119,000 times as of August 2026 — among the most-cited machine-learning papers ever.

Technically, the rise of generative models after ChatGPT (November 2022) reduced BERT's visible footprint. Even so, wherever the job is to read a piece of text and turn it into a vector — retrieval, classification, embedding generation — a 340M-parameter encoder remains a more sensible choice than a hundred-billion-parameter generator.

Pretrain-then-fine-tune; a Transformer-based general language model; open weights as the engine that accelerates a whole community — the premises of modern AI all run on rails BERT laid in 2018.

Four years before ChatGPT astonished the public, the NLP research field had already switched paradigms. BERT was that switch.