GPT-3 Announced

GPT-3 is the 175-billion-parameter autoregressive language model OpenAI published in 2020. It kept GPT-2's recipe, predicting the next token of text one step at a time, and scaled it by more than two orders of magnitude. The result could take on a wide range of language tasks with no task-specific training, from nothing more than a few examples placed in the prompt. What set it apart from earlier research models, though, was that nobody could download it. The weights stayed with OpenAI and the model was reached through an API, a way of releasing large models that became common afterwards.

Eight Models, One Name

"GPT-3" usually means the 175-billion-parameter model, but the paper actually trained eight models of different sizes. Measuring how capability grows with scale was the point of the exercise.

Every model had a 2,048-token context window. The architecture followed GPT-2: a stack of decoder blocks from the Transformer. The paper describes 175 billion parameters as "10x more than any previous non-sparse language model"; the largest it cites before that is Microsoft's Turing-NLG at 17 billion.

Three Hundred Billion Tokens

All eight models were trained on a total of 300 billion tokens, and the mixture was deliberately skewed.

Common Crawl, by far the largest source, was not even read once (0.44 epochs); Wikipedia, judged higher quality, was seen 3.4 times. Appendix D of the paper puts the compute for the 175B model at about 3.14 × 10²³ FLOPs, or 3,640 petaflop/s-days. Training ran on V100 GPUs on part of a high-bandwidth cluster provided by Microsoft.

What "Few-Shot" Measured

The "few-shot" in the title is a description of how the model was evaluated. The paper separated three settings:

• Zero-shot: only a natural-language description of the task
• One-shot: the description plus one worked example
• Few-shot: as many examples as fit in the context window, typically ten to a hundred

In none of them were the model's weights updated. The examples were simply part of the input text. Even so, GPT-3 improved with scale on translation, question answering, cloze tasks, three-digit arithmetic and unscrambling words, and on some benchmarks it came close to the best results of models fine-tuned for the task.

The paper was candid about the gaps. Few-shot performance stayed weak on natural-language inference, such as the ANLI dataset, and on some reading-comprehension sets such as RACE and QuAC. A full section examined whether benchmark questions had leaked into the web-scraped training data.

People Could Not Tell the Articles Apart

One of the most cited experiments asked people to read news articles of about 200 words and judge whether a machine had written them.

Chance is 50%, so the 175B model's articles were barely distinguishable from human writing. The paper connects this result to its section on the misuse of language models.

An API Instead of the Weights

How GPT-3 was released follows from what had happened with GPT-2 the year before. On 14 February 2019 OpenAI announced GPT-2 (1.5 billion parameters) but withheld the trained model, citing concerns about malicious use, and published a much smaller one instead. The full 1.5B weights came out on 5 November 2019, at the end of a staged release.

GPT-3's weights were never released. On 11 June 2020 OpenAI instead announced an API offering a general-purpose "text in, text out" interface and opened a waitlist for a private beta. The announcement said the API ran models with weights from the GPT-3 family, and named Algolia, Quizlet and Reddit, along with researchers at the Middlebury Institute, among those already using it.

Why not open-source it? The announcement's FAQ gave three reasons. Commercialising the technology would pay for OpenAI's research, safety and policy work. The models were large, expensive and difficult to deploy, which put them out of reach of anyone but large companies. And an API let OpenAI restrict access later if misuse appeared, which released weights would not.

On 22 September 2020 Microsoft announced that it had taken an exclusive licence to GPT-3. The same announcement said OpenAI would continue to offer GPT-3 through its own API, so outside developers still came in through the same door. The waitlist was removed on 18 November 2021; by then the API also offered Instruct-series models tuned to follow instructions, and Codex, which translated natural language into code.

Recognition, and What Came After

In December 2020 the paper was named one of three Best Paper Award winners at NeurIPS 2020. Written by 31 authors and running to 75 pages with appendices, it did not present a new algorithm. What it offered was strong evidence for an empirical rule: make the model larger, and it can do more without task-specific training.

The descendants have their own pages. Codex, trained further on code, became the engine of GitHub Copilot in 2021, and ChatGPT, a GPT-3.5-series model tuned for dialogue, reached the general public in November 2022. Almost nobody ever ran GPT-3 on their own hardware. The habit of leaving the model behind an API and calling it with a prompt started here, in June 2020.