Developing a computer use model
开发计算机使用模型

Claude can now use computers. The latest version of Claude 3.5 Sonnet can, when run through the appropriate software setup, follow a user’s commands to move a cursor around their computer’s screen, click on relevant locations, and input information via a virtual keyboard, emulating the way people interact with their own computer.
克劳德现在可以使用计算机。克劳德 3.5 诗篇的最新版本可以在通过适当的软件设置运行时，按照用户的命令在计算机屏幕上移动光标，点击相关位置，并通过虚拟键盘输入信息，模拟人们与自己计算机的互动方式。

We think this skill—which is currently in public beta—represents a significant breakthrough in AI progress. Below, we share some insights from the research that went into developing computer use models—and into making them safer.
我们认为这项技能——目前处于公开测试阶段——代表了人工智能进步的重大突破。下面，我们分享一些关于开发计算机使用模型的研究见解，以及如何使它们更安全。

Why computer use? 为什么使用计算机？

Why is this new capability important? A vast amount of modern work happens via computers. Enabling AIs to interact directly with computer software in the same way people do will unlock a huge range of applications that simply aren’t possible for the current generation of AI assistants.
为什么这个新功能很重要？现代工作的绝大部分都是通过计算机进行的。让人工智能以与人类相同的方式直接与计算机软件互动，将解锁一系列当前一代人工智能助手无法实现的应用。

Over the last few years, many important milestones have been reached in the development of powerful AI—for example, the ability to perform complex logical reasoning and the ability to see and understand images. The next frontier is computer use: AI models that don’t have to interact via bespoke tools, but that instead are empowered to use essentially any piece of software as instructed.
在过去几年中，强大人工智能的发展取得了许多重要里程碑——例如，进行复杂逻辑推理的能力和看懂图像的能力。下一个前沿是计算机使用：不需要通过定制工具进行交互的人工智能模型，而是能够根据指示使用任何软件。

The research process 研究过程

Our previous work on tool use and multimodality provided the groundwork for these new computer use skills. Operating computers involves the ability to see and interpret images—in this case, images of a computer screen. It also requires reasoning about how and when to carry out specific operations in response to what’s on the screen. Combining these abilities, we trained Claude to interpret what’s happening on a screen and then use the software tools available to carry out tasks.
我们之前关于工具使用和多模态的研究为这些新的计算机使用技能奠定了基础。操作计算机需要能够看到并解释图像——在这种情况下，是计算机屏幕的图像。它还需要推理如何以及何时根据屏幕上的内容执行特定操作。结合这些能力，我们训练了 Claude 来解释屏幕上发生的事情，然后使用可用的软件工具来完成任务。

When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place. Training Claude to count pixels accurately was critical. Without this skill, the model finds it difficult to give mouse commands—similar to how models often struggle with simple-seeming questions like “how many A’s in the word ‘banana’?”.
当开发者让 Claude 使用一款计算机软件并给予必要的访问权限时，Claude 会查看用户可见的屏幕截图，然后计算光标需要在垂直或水平方向上移动多少像素，以便在正确的位置点击。训练 Claude 准确地计算像素是至关重要的。没有这个技能，模型很难给出鼠标命令——这类似于模型在处理“‘香蕉’这个词中有多少个 A？”这样看似简单的问题时常常感到困难。

We were surprised by how rapidly Claude generalized from the computer-use training we gave it on just a few pieces of simple software, such as a calculator and a text editor (for safety reasons we did not allow the model to access the internet during training). In combination with Claude’s other skills, this training granted it the remarkable ability to turn a user’s written prompt into a sequence of logical steps and then take actions on the computer. We observed that the model would even self-correct and retry tasks when it encountered obstacles.
我们对克劳德从我们提供的仅仅几款简单软件（如计算器和文本编辑器）的计算机使用培训中迅速概括的能力感到惊讶（出于安全原因，我们在培训期间不允许模型访问互联网）。结合克劳德的其他技能，这种培训赋予了它将用户的书面提示转化为一系列逻辑步骤并在计算机上采取行动的非凡能力。我们观察到，当模型遇到障碍时，它甚至会自我纠正并重试任务。

Although the subsequent advances came quickly once we made the initial breakthrough, it took a great deal of trial and error to get there. Some of our researchers noted that developing computer use was close to the “idealized” process of AI research they’d pictured when they first started in the field: constant iteration and repeated visits back to the drawing board until there was progress.
尽管在我们取得初步突破后，随后的进展迅速，但到达那里经历了大量的反复试验。一些研究人员指出，计算机使用的发展与他们最初进入该领域时想象的“理想化”人工智能研究过程非常接近：不断迭代和反复回到起点，直到取得进展。

The research paid off. At present, Claude is state-of-the-art for models that use computers in the same way as a person does—that is, from looking at the screen and taking actions in response. On one evaluation created to test developers’ attempts to have models use computers, OSWorld, Claude currently gets 14.9%. That’s nowhere near human-level skill (which is generally 70-75%), but it’s far higher than the 7.7% obtained by the next-best AI model in the same category.
研究得到了回报。目前，Claude 是使用计算机的模型中最先进的，方式与人类相同——即通过观察屏幕并做出相应的行动。在一个旨在测试开发者让模型使用计算机的评估中，OSWorld，Claude 目前得分为 14.9%。这远未达到人类水平的技能（通常为 70-75%），但远高于同类中下一个最佳 AI 模型获得的 7.7%。

Making computer use safe 确保计算机使用安全

Every advance in AI brings with it new safety challenges. Computer use is mainly a way of lowering the barrier to AI systems applying their existing cognitive skills, rather than fundamentally increasing those skills, so our chief concerns with computer use focus on present-day harms rather than future ones. We confirmed this by assessing whether computer use increases the risk of frontier threats as outlined in our Responsible Scaling Policy. We found that the updated Claude 3.5 Sonnet, including its new computer use skill, remains at AI Safety Level 2—that is, it doesn’t require a higher standard of safety and security measures than those we currently have in place.
每一次人工智能的进步都带来了新的安全挑战。计算机的使用主要是降低人工智能系统应用其现有认知技能的门槛，而不是从根本上提高这些技能，因此我们对计算机使用的主要关注点集中在当前的危害上，而不是未来的危害。我们通过评估计算机使用是否增加了我们在负责任扩展政策中概述的前沿威胁的风险来确认这一点。我们发现，更新后的 Claude 3.5 Sonnet，包括其新的计算机使用技能，仍然保持在人工智能安全等级 2——也就是说，它不需要比我们目前所采取的安全和保障措施更高的标准。

When future models require AI Safety Level 3 or 4 safeguards because they present catastrophic risks, computer use might exacerbate those risks. We judge that it’s likely better to introduce computer use now, while models still only need AI Safety Level 2 safeguards. This means we can begin grappling with any safety issues before the stakes are too high, rather than adding computer use capabilities for the first time into a model with much more serious risks.
当未来的模型因存在灾难性风险而需要 AI 安全等级 3 或 4 的保护措施时，计算机使用可能会加剧这些风险。我们认为，现在引入计算机使用可能更好，因为模型仍然只需要 AI 安全等级 2 的保护措施。这意味着我们可以在风险尚未过高之前开始处理任何安全问题，而不是在具有更严重风险的模型中首次添加计算机使用能力。

In this spirit, our Trust & Safety teams have conducted extensive analysis of our new computer-use models to identify potential vulnerabilities. One concern they've identified is “prompt injection”—a type of cyberattack where malicious instructions are fed to an AI model, causing it to either override its prior directions or perform unintended actions that deviate from the user's original intent. Since Claude can interpret screenshots from computers connected to the internet, it’s possible that it may be exposed to content that includes prompt injection attacks.
在这种精神下，我们的信任与安全团队对我们的新计算机使用模型进行了广泛分析，以识别潜在的漏洞。他们识别出的一个问题是“提示注入”——一种网络攻击类型，其中恶意指令被输入到 AI 模型中，导致其覆盖先前的指令或执行偏离用户原始意图的意外操作。由于 Claude 可以解释连接到互联网的计算机的屏幕截图，因此它可能会接触到包含提示注入攻击的内容。

Those using the computer-use version of Claude in our public beta should take the relevant precautions to minimize these kinds of risks. As a resource for developers, we have provided further guidance in our reference implementation.
在我们的公共测试版中使用计算机版本的 Claude 的用户应采取相关预防措施，以最小化这些风险。作为开发者的资源，我们在我们的参考实现中提供了进一步的指导。

As with any AI capability, there’s also the potential for users to intentionally misuse Claude’s computer skills. Our teams have developed classifiers and other methods to flag and mitigate these kinds of abuses. Given the upcoming U.S. elections, we’re on high alert for attempted misuses that could be perceived as undermining public trust in electoral processes. While computer use is not sufficiently advanced or capable of operating at a scale that would present heightened risks relative to existing capabilities, we've put in place measures to monitor when Claude is asked to engage in election-related activity, as well as systems for nudging Claude away from activities like generating and posting content on social media, registering web domains, or interacting with government websites. We will continuously evaluate and iterate on these safety measures to balance Claude's capabilities with responsible use during the public beta.
与任何人工智能能力一样，用户也有可能故意滥用 Claude 的计算机技能。我们的团队开发了分类器和其他方法来标记和减轻这些滥用行为。鉴于即将到来的美国选举，我们对可能被视为破坏公众对选举过程信任的滥用行为保持高度警惕。虽然计算机使用尚未足够先进或具备在相对于现有能力的规模上操作的能力，但我们已采取措施监控 Claude 被要求参与与选举相关的活动，以及系统以引导 Claude 远离生成和发布社交媒体内容、注册网络域名或与政府网站互动等活动。我们将持续评估和迭代这些安全措施，以在公共测试期间平衡 Claude 的能力与负责任的使用。

Consistent with our standard approach to data privacy, by default we don’t train our generative AI models on user-submitted data, including any of the screenshots Claude receives.
根据我们对数据隐私的标准做法，默认情况下，我们不会在用户提交的数据上训练我们的生成性人工智能模型，包括 Claude 收到的任何截图。

The future of computer use
计算机使用的未来

Computer use is a completely different approach to AI development. Up until now, LLM developers have made tools fit the model, producing custom environments where AIs use specially-designed tools to complete various tasks. Now, we can make the model fit the tools—Claude can fit into the computer environments we all use every day. Our goal is for Claude to take pre-existing pieces of computer software and simply use them as a person would.
计算机使用是人工智能开发的完全不同的方法。直到现在，LLM 开发者一直在 使工具适应模型，创建定制环境，让人工智能使用专门设计的工具来完成各种任务。现在，我们可以 使模型适应工具——Claude 可以融入我们每天使用的计算机环境。我们的目标是让 Claude 使用现有的计算机软件，就像人一样。

There’s still a lot to do. Even though it’s the current state of the art, Claude’s computer use remains slow and often error-prone. There are many actions that people routinely do with computers (dragging, zooming, and so on) that Claude can’t yet attempt. The “flipbook” nature of Claude’s view of the screen—taking screenshots and piecing them together, rather than observing a more granular video stream—means that it can miss short-lived actions or notifications.
还有很多事情要做。尽管这是目前的尖端技术，Claude 的计算机使用仍然缓慢且常常出错。人们常规使用计算机的许多操作（拖动、缩放等）Claude 还无法尝试。Claude 对屏幕的“翻书”式视图——截取屏幕截图并将其拼接在一起，而不是观察更细粒度的视频流——意味着它可能会错过短暂的操作或通知。

Even while we were recording demonstrations of computer use for today’s launch, we encountered some amusing errors. In one, Claude accidentally clicked to stop a long-running screen recording, causing all footage to be lost. In another, Claude suddenly took a break from our coding demo and began to peruse photos of Yellowstone National Park.
即使在我们为今天的发布录制计算机使用演示时，我们也遇到了一些 有趣的错误。其中一次，Claude 不小心点击停止了一个长时间运行的屏幕录制，导致所有录像都丢失了。在另一次中，Claude 突然暂停了我们的编码演示，开始浏览黄石国家公园的照片。

We expect that computer use will rapidly improve to become faster, more reliable, and more useful for the tasks our users want to complete. It’ll also become much easier to implement for those with less software-development experience. At every stage, our researchers will be working closely with our safety teams to ensure that Claude’s new capabilities are accompanied by the appropriate safety measures.
我们预计计算机使用将迅速改善，变得更快、更可靠，并更有助于用户希望完成的任务。对于那些软件开发经验较少的人来说，实施也将变得更加容易。在每个阶段，我们的研究人员将与安全团队紧密合作，以确保 Claude 的新功能伴随适当的安全措施。

We invite developers who try computer use in our public beta to contact us with their feedback using this form, so that our researchers can continue to improve the usefulness and safety of this new capability.
我们邀请在我们的公共测试中尝试计算机使用的开发者通过使用此表单与我们联系，提供他们的反馈，以便我们的研究人员能够继续提高这一新功能的实用性和安全性。