Claude Computer Use — An AI That Looks at the Screen and Moves the Mouse

On 22 October 2024 Anthropic released computer use as a public beta, alongside an upgraded Claude 3.5 Sonnet. When a developer directs it, Claude looks at screenshots of the screen, moves a cursor, clicks buttons and types text. In other words, it can operate software that has no dedicated API, as long as there is a screen built for people. It was available through the Anthropic API, Amazon Bedrock and Google Cloud's Vertex AI.

If you want a one-line answer to "what is computer use", it is a way of giving an AI the same screen, mouse and keyboard a person uses. Until then, getting an AI to operate software meant wiring up an API for each operation; computer use replaced that wiring with the screen as a common entry point. Anthropic's own assessment at launch was modest: the announcement called the feature experimental, "at times cumbersome and error-prone."

How it works

Claude does not see the screen as continuous video but as screenshots taken at intervals. Anthropic likened this to a flipbook and explained that brief notifications or movements between frames can be missed. The key to development, it said, was training the model to count pixels so that it could say precisely where on the screen to move the cursor.

Developers mostly tried it in a sample environment Anthropic provided as a Docker container. The developer Simon Willison tried it on launch day and reported that a request to find pelican photos on his website worked first time, while Sudoku defeated it.

OSWorld is a benchmark that has an AI carry out tasks on real operating systems such as Ubuntu, Windows and macOS. According to OpenAI's materials, human performance on it is 72.4%, so the gap at launch was wide. On the same day Anthropic reported that Claude 3.5 Sonnet's score on the coding benchmark SWE-bench Verified rose from 33.4% to 49.0%.

What the failures showed

In its account of development Anthropic also published failures from recording demos. In one, "Claude accidentally clicked to stop a long-running screen recording, causing all footage to be lost." In another, it broke off a coding demo and started browsing photos of Yellowstone National Park. A screen built for people turned out to be distracting for an AI too.

The central worry: prompt injection

An AI that reads the screen can also obey what is written on it. From launch Anthropic warned about prompt injection, in which malicious instructions embedded in on-screen content override the user's, and urged developers to take precautions. Willison pointed to a researcher's demonstration that a single line on a web page could get Claude to download and run a file, and advised isolating Claude from sensitive data and actions.

Anthropic was also wary of election misuse. It said it monitored election-related requests and had built classifiers and other methods to steer Claude away from activities such as posting on social media or interacting with government websites.

The worry has not gone away. On 25 August 2025 Anthropic announced a pilot of a Chrome extension that runs Claude inside the browser, with 1,000 Max plan users. It reported that browser use without its safety mitigations had a 23.6% attack success rate across 123 test cases, and that the mitigations brought this down to 11.2% in autonomous mode. That halved the rate but did not eliminate it. In the same post the company wrote that it viewed browser-using AI as inevitable.

How it improved

About three months later OpenAI shipped the same idea to consumers as Operator, and screen-operating AI went from one company's experiment to a field of competition. OSWorld scores more than quadrupled in a year and moved towards the human level. All of these figures were published by the developers themselves.

What it changed

Computer use broke the assumption that before you can hand work to an AI you must first build it an entry point. Many business systems have no API and assume a person working through a screen; an AI that can read the screen can walk straight into those older systems. The flip side is that an AI running in a user's browser or PC sees the same malicious pages the user sees. Convenience and risk arrive through the same door, and that has remained the defining trait of the technology.

The same idea applied to code rather than screens became Claude Code, and the standard for connecting to outside tools is MCP. The product of Cognition, one of the early testers, is on the Devin page. The surrounding events are on the AI timeline.