Anthropics Cutting-Edge AI Puts Your PC on Autopilot

In a bold pitch to investors last spring, Anthropic outlined its vision for AI that could revolutionize virtual assistants. These assistants would be capable of conducting research, responding to emails, and managing back-office tasks autonomously. Anthropic described this breakthrough as a “next-gen algorithm for AI self-teaching” — a powerful tool they believed could one day automate vast sectors of the economy.

That future is now unfolding. Anthropic has unveiled an upgraded version of its Claude 3.5 Sonnet model, equipped to understand and interact with any desktop application. Through the new “Computer Use” API, currently in open beta, the model can emulate keystrokes, button clicks, and mouse gestures, mimicking the actions of a real user on a PC.

“We trained Claude to interpret what’s happening on a screen and leverage available software tools to execute tasks,” Anthropic shared in a blog post. When tasked with operating a program, Claude evaluates screenshots and determines cursor movements and actions with precision, allowing it to click, type, and scroll as needed. Developers can test the Computer Use API via Anthropic’s platform, Amazon Bedrock, and Google Cloud’s Vertex AI. The new 3.5 Sonnet, even without Computer Use, is being integrated into Claude apps, offering substantial performance upgrades over its predecessor.

Automating Tasks

Automating tasks on a PC isn’t a groundbreaking concept — companies ranging from long-established RPA vendors to emerging startups like Relay, Induced AI, and Automat have been tackling this challenge for years. But the race to develop “AI agents” is heating up. AI agents, designed to automate software processes, are increasingly seen as a lucrative pathway for companies seeking to monetize their AI investments. According to Capgemini, 10% of organizations are already using AI agents, and 82% expect to integrate them within the next three years.

Anthropic’s take on AI agents introduces an “action-execution layer” that empowers the 3.5 Sonnet to perform desktop-level commands. Notably, it can now browse the web and use any application, marking a significant first for Anthropic.

What sets Anthropic’s 3.5 Sonnet apart from the other AI agents on the market? It’s a fair question. While companies like Rabbit are developing web agents that can handle tasks such as purchasing movie tickets, and Adept, recently acquired by Amazon, focuses on training models to browse and interact with software, others like Twin Labs are leveraging existing models like OpenAI’s GPT-4o to automate desktop processes.

Anthropic, however, asserts that the 3.5 Sonnet is a far more powerful and robust model. According to the SWE-bench Verified benchmark, it excels in coding tasks, surpassing even OpenAI’s flagship o1. The upgraded 3.5 Sonnet not only self-corrects when encountering obstacles but also tackles complex objectives involving dozens or even hundreds of steps.

Hold off on replacing your secretary just yet.

The model completed less than half of the tasks in a recent evaluation that tested the 3.5 Sonnet’s ability to handle airline booking tasks, such as modifying flight reservations. Another test involving returns encountered challenges, failing about a third of the time.

Anthropic acknowledges that the upgraded 3.5 Sonnet struggles with fundamental actions like scrolling and zooming. It can also miss quick, transient actions and notifications due to the way it processes screenshots.

“Claude’s Computer Use remains slow and prone to errors,” Anthropic noted in its post, encouraging developers to start with low-risk tasks as they explore the tool’s potential.

Risky Business

Is the new 3.5 Sonnet powerful enough to pose risks? Possibly.

A recent study revealed that models without desktop app capabilities, like OpenAI’s GPT-4o, were susceptible to engaging in harmful “multi-step agent behavior,” such as attempting to order a fake passport on the dark web when targeted by jailbreaking techniques. According to researchers, even models protected by filters could perform harmful tasks when compromised.

With desktop access, a model could cause more significant disruption by exploiting app vulnerabilities or mishandling personal information. The model’s ability to connect online and interact with applications could provide opportunities for malicious actors.

Anthropic acknowledges the risks but argues that the potential benefits of studying the model’s real-world use outweigh the dangers.

“It’s far better to grant access to computers through today’s more limited, safer models,” the company wrote. “This approach allows us to monitor and address any issues at this early stage, while progressively building up computer-use capabilities and safety measures.”

Anthropic has implemented strong safeguards to prevent misuse of the new 3.5 Sonnet. They’ve ensured that the model was not trained on user screenshots or prompts and have blocked its web access during training. The company has also developed classifiers to steer 3.5 Sonnet away from high-risk actions, such as posting on social media, creating accounts, or interacting with government websites.

With the U.S. general election approaching, Anthropic is focused on preventing election-related abuse. The U.S. AI Safety Institute and the U.K. Safety Institute, government agencies tasked with evaluating AI model risks, tested the 3.5 Sonnet before its launch.

Anthropic assured TechCrunch that it can restrict access to additional websites and features, if necessary, to guard against threats like spam, fraud, or misinformation. As a precaution, the company retains screenshots captured by Computer Use for at least 30 days, though this retention period may raise concerns for some developers.

When asked under what conditions they would hand screenshots to third parties, such as law enforcement, Anthropic confirmed they would comply with requests through valid legal processes.

“There are no foolproof methods, and we will continuously evaluate and enhance our safety measures to balance Claude’s capabilities with responsible use,” the company stated. Users are advised to take precautions to minimize risks, including isolating Claude from sensitive computer data.

These measures are designed to ensure that the worst-case scenarios are avoided.

A Cheaper Model

The spotlight may be on the upgraded 3.5 Sonnet model, but Anthropic has also revealed exciting news about an upcoming update to its Claude series. The new 3.5 Haiku model, the most cost-effective and efficient in the lineup, is set to launch soon.

The Claude 3.5 Haiku, arriving in the coming weeks, will deliver performance on par with the former flagship, Claude 3 Opus, at the exact cost and with the “approximate speed” of the current Claude 3 Haiku.

“With low latency, enhanced instruction following, and improved tool use, Claude 3.5 Haiku is ideal for user-facing applications, specialized sub-agent tasks, and creating personalized experiences from massive data sets—such as purchase histories, pricing, or inventory data,” Anthropic wrote in a blog post.

Initially released as a text-only model, 3.5 Haiku will evolve into a multimodal version capable of analyzing text and images.

Does this leave much room for using 3 Opus or the upcoming 3.5 Opus, which was hinted at back in June? According to an Anthropic spokesperson, every model in the Claude 3 family has a unique value for different customer needs.

“Claude 3.5 Opus is on our roadmap, and we’ll share more details as soon as possible,” the spokesperson confirmed.