Anthropic Gives Claude 3.5 Sonnet the Ability to Use a Computer
The upgraded model can view screens, move cursors, and execute tasks across applications without API integrations.
Anthropic has equipped its Claude 3.5 Sonnet model with a new capability that lets the AI interact with computers the way humans do. The upgrade, available in public beta through the Anthropic API, Amazon Bedrock, and Google Cloud's Vertex AI, marks a significant shift in how artificial intelligence agents operate within digital environments.
What the capability does
Claude can now view a screen, move a cursor, click buttons, and type text to execute complex tasks across multiple applications. The technology works by having the model analyze screenshots of the desktop and then send corresponding mouse and keyboard commands to interact with the system. This means the AI can navigate any software that a human user can access, without requiring developers to build specific API integrations for each tool.
Previously, AI agents relied on structured APIs to interact with software applications. That approach required custom integration work for every program an agent needed to use. Anthropic's method bypasses this limitation by using the same general-purpose interface that humans rely on: the visual display and input devices. The model sees what's on screen, decides what action to take, and executes it through standard operating system commands.
Why this matters for automation
This represents a fundamental shift toward 'Agentic AI'—systems that can operate autonomously across a user's entire digital workspace. End-to-end workflows spanning multiple applications could be automated without manual human navigation between different software tools. A single AI agent could theoretically open a browser, extract data from a webpage, paste it into a spreadsheet, run calculations, and email the results—all without specialized integrations for each step.
The move from API-dependent interactions to screen-based control expands the range of tasks AI can handle. Legacy software without modern APIs becomes accessible. Proprietary applications with restricted integration options can still be operated. The bottleneck shifts from technical integration work to the AI's ability to understand visual interfaces and execute appropriate actions.
What to watch next
The capability is currently in public beta, which means Anthropic is gathering real-world performance data before broader deployment. Key questions remain around reliability, speed, and safety when an AI system has direct control over a computer interface. Errors in cursor placement or keystroke execution could have unintended consequences in production environments.
Anthropic stated that 'Claude can now use a computer like a person does,' but the practical limits of that comparison will become clear as developers test the feature across diverse workflows. The company has not disclosed specific performance benchmarks or error rates for computer use tasks. Independent testing will determine whether the capability delivers on its promise of human-like interaction or requires significant human oversight to remain reliable.
The upgrade positions Claude 3.5 Sonnet against competing models that have pursued similar agentic capabilities. How this translates to enterprise adoption depends on whether the screen-based approach proves more flexible and reliable than API-centric alternatives in real-world deployments.