Anthropic launches AI model capable of controlling your PC

Anthropic launches AI model capable of controlling your PC
23 Oct 2024

Anthropic, a leading artificial intelligence (AI) company, has launched an upgraded version of its Claude 3.5 Sonnet model.

The innovative AI model can now interact with any desktop application via a newly introduced "Computer Use" API, which is currently in the open beta stage.

The upgraded model can mimic human actions like keystrokes, button clicks, and mouse gestures on a computer.

AI model's training and functionality
Training process

Anthropic has trained Claude to understand what's happening on-screen and use available software tools to perform tasks.

When a developer gives Claude a task involving certain software and provides necessary access, the AI model examines screenshots of what's visible to the user.

It then determines how many pixels it needs to move a cursor vertically or horizontally for precise clicking.

Accessing and testing the new AI model
User access

Developers can try out the Computer Use feature via Anthropic's API, Amazon Bedrock, or Google Cloud's Vertex AI platform.

The latest 3.5 Sonnet model without Computer Use is being rolled out to Claude apps, providing a range of performance improvements over its predecessor.

This update represents a major step in Anthropic's quest to build AI-powered virtual assistants capable of automating large parts of the economy.

AI model's web browsing capabilities and user control
Web interaction

Anthropic has added an "action-execution layer" in the new 3.5 Sonnet model, enabling it to execute desktop-level commands.

The upgraded model can browse the web and use any website/app, a first for Anthropic's AI models.

"Humans remain in control by providing specific prompts that direct Claude's actions," an Anthropic spokesperson told TechCrunch, stressing that users can enable or restrict access as needed.

Practical applications and performance of the AI model
Real-world usage

Software development platform Replit has used an early version of the new 3.5 Sonnet model to create an "autonomous verifier" that evaluates apps during their creation phase.

Canva is also looking into possible ways to leverage the new model in aiding its design and editing process.

Despite its sophistication, Anthropic admits the upgraded 3.5 Sonnet still struggles with basic actions like scrolling and zooming, and can miss transient actions and notifications due to its screenshot-based approach.