Stable Diffusion Goes Public — Image Generation on Your Own GPU

Stable Diffusion is a generative model that turns a text prompt into an image, and on 22 August 2022 Stability AI released its weights to the public. Until then, image generators such as DALL·E and Midjourney were reachable only as cloud services: you sent a prompt and got a picture back. Stable Diffusion shipped the model itself, so anyone with a capable GPU could run it, modify it and train it further. Put simply, it was the release that let people take image-generating AI home — and in doing so it pushed questions about training data and copyright out of the lab and into courts and legislatures.

Who made it — CompVis, Stability AI, Runway, LAION

The core technique is the latent diffusion model developed by the CompVis group at LMU Munich. The paper by Robin Rombach and colleagues appeared on arXiv in December 2021 and was presented at CVPR 2022. Rather than removing noise from a full-resolution image step by step, it does so in a compressed "latent" space, which cuts the computation sharply.

The CompVis repository on GitHub says Stable Diffusion was made possible by a collaboration with Stability AI and Runway, with a compute donation from Stability AI and support from LAION.

In its announcement Stability AI described the licence as permissive for both commercial and non-commercial use and left responsible use to the user. It shipped an AI-based safety classifier, on by default, to remove unwanted outputs, and noted that its parameters could be adjusted.

What it meant to hand out the weights

With DALL·E or Midjourney, the provider decided what could be generated and the model stayed out of reach. Stable Diffusion moved that boundary to the user.

The effect was fast. On 17 October 2022 Stability AI announced US$101 million in funding and claimed that Stable Diffusion had been downloaded and licensed by more than 200,000 developers worldwide, and that its own web service, DreamStudio, had over a million registered users in more than 50 countries — the company's figures. Open-source front ends and fine-tuning techniques for teaching the model a particular style or subject appeared in quick succession, and a culture of sharing derivative models grew around them. A retrained Stable Diffusion 2.0 followed in November 2022, and SDXL in July 2023.

The release was an early instance of the open-weights path later taken by Llama. What ChatGPT did for text that November, Stable Diffusion had done for images a few months earlier.

Who gained, and who said they lost

The first beneficiaries were individual developers and small companies, who could build image generation into their own products without API fees or a provider's content policy. Researchers gained a capable model whose internals they could inspect.

The loudest objections came from people who make images for a living. In January 2023 three visual artists — Sarah Andersen, Kelly McKernan and Karla Ortiz — filed a class action in the Northern District of California against Stability AI, Midjourney and DeviantArt, alleging that their work had been used for training without permission. On 12 August 2024 Judge William Orrick allowed the direct and induced copyright infringement claims to proceed and dismissed the claims under the DMCA's provisions on copyright management information.

Photography followed. Getty Images sued Stability AI in England in January 2023 and in the United States in February; that dispute and what the English court decided are covered in Getty Images v. Stability AI. Whether an AI-made image can itself be copyrighted was being tested the same summer, when an image made with Midjourney won a prize at the Colorado State Fair.

LAION-5B — what was in the training data

LAION-5B, the source of Stable Diffusion's training images, was assembled from the web by LAION, a German non-profit. By LAION's own description it holds about 5.8 billion image links paired with text. It is a list of links rather than a store of images, but far too large for people to review by hand.

On 20 December 2023 David Thiel of the Stanford Internet Observatory published a report finding many hundreds of instances of known child sexual abuse material (CSAM) in LAION-5B; the report notes that parts of the dataset were used to train Stable Diffusion. LAION took the dataset down immediately. On 30 August 2024 it released Re-LAION-5B, with 2,236 links removed after matching against link and image hash lists supplied by partner organisations. According to LAION, Stanford had identified 1,008 links.

In Japan

Japan's Copyright Act, through flexible limitations introduced in the 2018 amendment (Article 30-4 and related provisions), broadly permits the analysis of works for purposes other than enjoying their expression. After Stable Diffusion appeared, how far that provision covers generative AI training became a live question, and in March 2024 the legal subcommittee of the Copyright Subdivision of the Council for Cultural Affairs adopted a document setting out its general understanding of AI and copyright. It is an administrative interpretation, not a court ruling.

Afterwards

By putting image generation on ordinary hardware, Stable Diffusion widened the circle of people using generative AI almost overnight, and sent three questions — consent for training, the safety of training data, and payment to creators — to courts and parliaments. How the same questions are being fought over for text is told in Bartz v. Anthropic and The New York Times v. OpenAI.