Developing Computer Use

Anthropic trained Claude 3.5 Sonnet to operate software by interpreting screenshots and issuing mouse and keyboard commands. Its October 2024 account says that when a developer gives Claude access to a piece of software, Claude looks at screenshots, counts how many pixels to move the cursor to click the right place, and types through a virtual keyboard. At the time it was in public beta, and the approach aims to let a model use existing software as a person would instead of needing a custom tool for each task.

That screenshot-based view also constrained the launch version. Anthropic describes the system as slow and often error-prone, and says Claude's "flipbook" view of the screen means it can miss short-lived actions or notifications between screenshots. This was an early capability with clear limits on dependable task completion.