Agent AI:
Surveying the Horizons of Multimodal Interaction
代理AI： 多模态相互作用的视野

Agents that generate text are often prone to hallucinations, which are instances where the generated text is nonsensical or unfaithful to the provided source content Raunak et al. (2021); Maynez et al. (2020). Hallucinations can be split into two categories, intrinsic and extrinsic Ji et al. (2023). Intrinsic hallucinations are hallucinations that are contradictory to the source material, whereas extrinsic hallucinations are when the generated text contains additional information that was not originally included in the source material.
生成文本的代理通常容易产生幻觉，这是生成的文本对所提供的源内容无意义或不忠实的情况Raunak et al.（2021）; Maynez et al.（2020）。幻觉可以分为两类，内在和外在Ji et al.（2023）。内在幻觉是与原始材料相矛盾的幻觉，而外在幻觉是当生成的文本包含原始材料中最初没有包含的额外信息时。

Some promising routes for reducing the rate of hallucination in language generation involve using retrieval-augmented generation Lewis et al. (2020); Shuster et al. (2021) or other methods for grounding natural language outputs via external knowledge retrieval Dziri et al. (2021); Peng et al. (2023). Generally, these methods seek to augment language generation by retrieving additional source material and by providing mechanisms to check for contradictions between the generated response and the source material.
降低语言生成中幻觉发生率的一些有希望的方法包括使用检索增强生成刘易斯等人（2020）;舒斯特等人（2021）或其他通过外部知识检索来建立自然语言输出的方法。通常，这些方法通过检索附加的源材料并通过提供检查所生成的响应与源材料之间的矛盾的机制来寻求增强语言生成。

Within the context of multi-modal agent systems, VLMs have been shown to hallucinate as well Zhou et al. (2023b). One common cause of hallucination for vision-based language-generation is due to the over-reliance on co-occurrence of objects and visual cues in the training data Rohrbach et al. (2018). AI agents that exclusively rely upon pretrained LLMs or VLMs and use limited environment-specific finetuning can be particularly vulnerable to hallucinations since they rely upon the internal knowledge-base of the pretrained models for generating actions and may not accurately understand the dynamics of the world state in which they are deployed.
在多模式代理系统的背景下，VLM也被证明会产生幻觉Zhou等人（2023 b）。基于视觉的语言生成产生幻觉的一个常见原因是过度依赖训练数据中物体和视觉线索的共同出现Rohrbach et al.（2018）。完全依赖于预训练的LLMs或VLM并使用有限的环境特定微调的AI代理可能特别容易受到幻觉的影响，因为它们依赖于预训练模型的内部知识库来生成动作，并且可能无法准确理解它们所部署的世界状态的动态。

AI agents based on LLMs or LMMs (large multimodal models) have biases due to several factors inherent in their design and training process. When designing these AI agents, we must be mindful of being inclusive and aware of the needs of all end users and stakeholders. In the context of AI agents, inclusivity refers to the measures and principles employed to ensure that the agent’s responses and interactions are inclusive, respectful, and sensitive to a wide range of users from diverse backgrounds. We list key aspects of agent biases and inclusivity below.
基于LLMs或Lectin（大型多模态模型）的AI代理由于其设计和训练过程中固有的几个因素而存在偏差。在设计这些人工智能代理时，我们必须注意包容性，并意识到所有最终用户和利益相关者的需求。在人工智能代理的背景下，包容性是指用于确保代理的响应和交互对来自不同背景的广泛用户具有包容性、尊重性和敏感性的措施和原则。我们在下面列出了代理人偏见和包容性的关键方面。

• •

  Training Data: Foundation models are trained on vast amounts of text data collected from the internet, including books, articles, websites, and other text sources. This data often reflects the biases present in human society, and the model can inadvertently learn and reproduce these biases. This includes stereotypes, prejudices, and slanted viewpoints related to race, gender, ethnicity, religion, and other personal attributes. In particular, by training on internet data and often only English text, models implicitly learn the cultural norms of Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies Henrich et al. (2010) who have a disproportionately large internet presence. However, it is essential to recognize that datasets created by humans cannot be entirely devoid of bias, since they frequently mirror the societal biases and the predispositions of the individuals who generated and/or compiled the data initially.
  训练数据：基础模型基于从互联网收集的大量文本数据进行训练，包括书籍，文章，网站和其他文本来源。这些数据通常反映了人类社会中存在的偏见，模型可能会无意中学习和复制这些偏见。这包括与种族、性别、民族、宗教和其他个人属性有关的成见、偏见和偏见。特别是，通过对互联网数据的训练，而且通常只对英语文本进行训练，模型隐含地学习了西方、教育、工业化、富裕和民主（WEIRD）社会的文化规范。 然而，必须认识到，人类创建的数据集不可能完全没有偏见，因为它们经常反映出最初生成和/或汇编数据的个人的社会偏见和倾向。

• •

  Historical and Cultural Biases: AI models are trained on large datasets sourced from diverse content. Thus, the training data often includes historical texts or materials from various cultures. In particular, training data from historical sources may contain offensive or derogatory language representing a particular society’s cultural norms, attitudes, and prejudices. This can lead to the model perpetuating outdated stereotypes or not fully understanding contemporary cultural shifts and nuances.
  历史和文化偏见：人工智能模型是在来自不同内容的大型数据集上训练的。因此，训练数据通常包括来自各种文化的历史文本或材料。特别是，来自历史来源的培训数据可能包含代表特定社会文化规范、态度和偏见的冒犯性或贬损性语言。这可能会导致模式延续过时的刻板印象或不完全理解当代文化的转变和细微差别。

• •

  Language and Context Limitations: Language models might struggle with understanding and accurately representing nuances in language, such as sarcasm, humor, or cultural references. This can lead to misinterpretations or biased responses in certain contexts. Furthermore, there are many aspects of spoken language that are not captured by pure text data, leading to a potential disconnect between human understanding of language and how models understand language.
  语言和上下文限制：语言模型可能难以理解和准确表示语言中的细微差别，例如讽刺、幽默或文化引用。在某些情况下，这会导致误解或有偏见的反应。此外，口语的许多方面无法通过纯文本数据捕获，导致人类对语言的理解与模型如何理解语言之间存在潜在的脱节。

• •

  Policies and Guidelines: AI agents operate under strict policies and guidelines to ensure fairness and inclusivity. For instance, in generating images, there are rules to diversify depictions of people, avoiding stereotypes related to race, gender, and other attributes.
  政策和准则：人工智能代理在严格的政策和准则下运作，以确保公平性和包容性。例如，在生成图像时，有规则来多样化对人的描述，避免与种族、性别和其他属性有关的刻板印象。

• •

  Overgeneralization: These models tend to generate responses based on patterns seen in the training data. This can lead to overgeneralizations, where the model might produce responses that seem to stereotype or make broad assumptions about certain groups.
  过度泛化：这些模型倾向于根据训练数据中看到的模式生成响应。这可能会导致过度概括，模型可能会产生似乎刻板印象或对某些群体做出广泛假设的反应。

• •

  Constant Monitoring and Updating: AI systems are continuously monitored and updated to address any emerging biases or inclusivity issues. Feedback from users and ongoing research in AI ethics play a crucial role in this process.
  持续监测和更新：人工智能系统会持续监测和更新，以解决任何新出现的偏见或包容性问题。来自用户的反馈和正在进行的人工智能伦理研究在这一过程中发挥着至关重要的作用。

• •

  Amplification of Dominant Views: Since the training data often includes more content from dominant cultures or groups, the model may be more biased towards these perspectives, potentially underrepresenting or misrepresenting minority viewpoints.
  主流观点的放大：由于训练数据通常包含更多来自主流文化或群体的内容，因此模型可能更偏向于这些观点，可能会低估或歪曲少数群体的观点。

• •

  Ethical and Inclusive Design: AI tools should be designed with ethical considerations and inclusivity as core principles. This includes respecting cultural differences, promoting diversity, and ensuring that the AI does not perpetuate harmful stereotypes.
  道德和包容性设计：人工智能工具的设计应将道德考虑和包容性作为核心原则。这包括尊重文化差异，促进多样性，并确保AI不会使有害的陈规定型观念永久化。

• •

  User Guidelines: Users are also guided on how to interact with AI in a manner that promotes inclusivity and respect. This includes refraining from requests that could lead to biased or inappropriate outputs. Furthermore, it can help mitigate models learning harmful material from user interactions.
  用户指南：还指导用户如何以促进包容性和尊重的方式与人工智能进行交互。这包括避免提出可能导致有偏见或不适当产出的要求。此外，它还可以帮助减少模型从用户交互中学习有害内容的情况。

Despite these measures, AI agents still exhibit biases. Ongoing efforts in agent AI research and development are focused on further reducing these biases and enhancing the inclusivity and fairness of agent AI systems. Efforts to Mitigate Biases:
尽管有这些措施，人工智能代理仍然表现出偏见。人工智能研究和开发的持续努力集中在进一步减少这些偏见，提高人工智能系统的包容性和公平性。减少偏见的努力：

• •

  Diverse and Inclusive Training Data: Efforts are made to include a more diverse and inclusive range of sources in the training data.
  多样化和包容性的培训数据：努力在培训数据中纳入更多样化和更具包容性的来源。

• •

  Bias Detection and Correction: Ongoing research focuses on detecting and correcting biases in model responses.
  偏差检测和纠正：正在进行的研究重点是检测和纠正模型响应中的偏差。

• •

  Ethical Guidelines and Policies: Models are often governed by ethical guidelines and policies designed to mitigate biases and ensure respectful and inclusive interactions.
  道德准则和政策：模型通常受旨在减少偏见并确保尊重和包容性互动的道德准则和政策的约束。

• •

  Diverse Representation: Ensuring that the content generated or the responses provided by the AI agent represent a wide range of human experiences, cultures, ethnicities, and identities. This is particularly relevant in scenarios like image generation or narrative construction.
  多样化的表示：确保AI代理生成的内容或提供的响应代表广泛的人类经验，文化，种族和身份。这在图像生成或叙事构建等场景中尤为重要。

• •

  Bias Mitigation: Actively working to reduce biases in the AI’s responses. This includes biases related to race, gender, age, disability, sexual orientation, and other personal characteristics. The goal is to provide fair and balanced responses that do not perpetuate stereotypes or prejudices.
  偏见缓解：积极努力减少AI反应中的偏见。这包括与种族、性别、年龄、残疾、性取向和其他个人特征相关的偏见。目标是提供公平和平衡的反应，不使陈规定型观念或偏见永久化。

• •

  Cultural Sensitivity: The AI is designed to be culturally sensitive, acknowledging and respecting the diversity of cultural norms, practices, and values. This includes understanding and appropriately responding to cultural references and nuances.
  文化敏感性：人工智能旨在对文化敏感，承认和尊重文化规范，实践和价值观的多样性。这包括理解和适当地应对文化参考和细微差别。

• •

  Accessibility: Ensuring that the AI agent is accessible to users with different abilities, including those with disabilities. This can involve incorporating features that make interactions easier for people with visual, auditory, motor, or cognitive impairments.
  可访问性：确保AI代理可供具有不同能力的用户访问，包括残疾人。这可能涉及整合功能，使视觉，听觉，运动或认知障碍的人更容易进行交互。

• •

  Language-based Inclusivity: Providing support for multiple languages and dialects to cater to a global user base, and being sensitive to the nuances and variations within a language Liu et al. (2023b).
  基于语言的包容性：提供对多种语言和方言的支持，以满足全球用户群的需求，并对语言中的细微差别和变化保持敏感。 

• •

  Ethical and Respectful Interactions: The Agent is programmed to interact ethically and respectfully with all users, avoiding responses that could be deemed offensive, harmful, or disrespectful.
  道德和尊重的互动：代理程序被编程为与所有用户进行道德和尊重的互动，避免可能被视为冒犯，有害或不尊重的反应。

• •

  User Feedback and Adaptation: Incorporating user feedback to continually improve the inclusivity and effectiveness of the AI agent. This includes learning from interactions to better understand and serve a diverse user base.
  用户反馈和自适应：收集用户反馈，以不断提高AI代理的包容性和有效性。这包括从互动中学习，以更好地理解和服务于多样化的用户群。

• •

  Compliance with Inclusivity Guidelines: Adhering to established guidelines and standards for inclusivity in AI agent, which are often set by industry groups, ethical boards, or regulatory bodies.
  遵守包容性准则：遵守人工智能代理的包容性准则和标准，这些准则和标准通常由行业组织、道德委员会或监管机构制定。

Despite these efforts, it’s important to be aware of the potential for biases in responses and to interpret them with critical thinking. Continuous improvements in AI agent technology and ethical practices aim to reduce these biases over time. One of the overarching goals for inclusivity in agent AI is to create an agent that is respectful and accessible to all users, regardless of their background or identity.
尽管有这些努力，重要的是要意识到反应中可能存在的偏见，并用批判性思维来解释它们。人工智能代理技术和道德实践的不断改进旨在随着时间的推移减少这些偏见。Agent AI包容性的首要目标之一是创建一个尊重所有用户的Agent，无论他们的背景或身份如何。

One key ethical consideration of AI agents involves comprehending how these systems handle, store, and potentially retrieve user data. We discuss key aspects below:
人工智能代理的一个关键道德考虑因素涉及理解这些系统如何处理，存储和潜在地检索用户数据。我们讨论以下关键方面：

The inference ability of an AI agent lies in its capacity to interpret, predict, and respond based on its training and input data. While these capabilities are advanced and continually improving, it’s important to recognize their limitations and the influence of the underlying data they are trained on. Particularly, in the context of large language models, it refers to its capacity to draw conclusions, make predictions, and generate responses based on the data it has been trained on and the input it receives. Inference augmentation in AI agents refers to enhancing the AI’s natural inference abilities with additional tools, techniques, or data to improve its performance, accuracy, and utility. This can be particularly important in complex decision-making scenarios or when dealing with nuanced or specialized content. We denote particularly important sources for inference augmentation below:
人工智能代理的推理能力在于它根据训练和输入数据解释，预测和响应的能力。虽然这些能力是先进的，并在不断改进，但重要的是要认识到它们的局限性和它们所训练的基础数据的影响。特别是在大型语言模型的背景下，它指的是根据它所训练的数据和接收的输入得出结论，做出预测和生成响应的能力。AI代理中的推理增强是指通过额外的工具，技术或数据来增强AI的自然推理能力，以提高其性能，准确性和实用性。这在复杂的决策场景中或处理细微差别或专业内容时尤为重要。我们在下面表示推理增强的特别重要的来源：

Recently, Agent AI has made significant advancements, and its integration into embodied systems has opened new possibilities for interacting with agents via more immersive, dynamic, and engaging experiences. To expedite the process and ease the cumbersome work in agent AI developing, we are proposing to develop the next-generation AI-empowered pipeline for agent interaction. Develop a human-machine collaboration system where humans and machines can communicate and interact meaningfully. The system can leverage the LLM’s or VLM dialog capabilities and vast action to talk with human players and identify human needs. Then it will perform proper actions to help human players upon request.
最近，Agent AI取得了重大进展，它与具体系统的集成为通过更具沉浸式，动态和吸引力的体验与代理进行交互开辟了新的可能性。为了加快这一过程并减轻Agent AI开发中的繁琐工作，我们建议为Agent交互开发下一代AI授权管道。开发一个人机协作系统，人类和机器可以进行有意义的交流和互动。系统可以利用LLM或VLM对话能力和大量动作来与人类玩家交谈并识别人类需求。然后，它将根据请求执行适当的操作以帮助人类玩家。

When employing LLM/VLMs for a human-machine collaboration system, it is essential to note that these operate as black boxes, generating unpredictable output. This uncertainty can become crucial in a physical setup, such as operating actual robotics. An approach to address this challenge is constraining the focus of the LLM/VLM through prompt engineering. For instance, in robotic task planning from instructions, providing environmental information within the prompt has been reported to yield more stable outputs than relying solely on text Gramopadhye and Szafir (2022). This report is supported by the Minsky’s frame theory of AI Minsky (1975), suggesting that the problem space to be solved by LLM/VLMs is defined by the given prompts. Another approach is designing prompts to make LLM/VLMs include explanatory text to allow users understand what the model has focused on or recognized. Additionally, implementing a higher layer that allows for pre-execution verification and modification under human guidance can facilitate the operation of systems working under such guidance (Fig. 4).
在人机协作系统中使用LLM/VLM时，必须注意这些操作就像黑盒一样，会产生不可预测的输出。这种不确定性在物理设置中可能变得至关重要，例如操作实际的机器人。解决这一挑战的一种方法是通过快速工程来限制LLM/VLM的重点。例如，在根据指令进行机器人任务规划时，据报道，在提示中提供环境信息比仅依赖文本Gramopadhye和Szafir（2022）产生更稳定的输出。该报告得到了AIMinsky（1975）的Minsky框架理论的支持，该理论认为LLM/VLMs要解决的问题空间由给定的提示定义。另一种方法是设计提示，使LLM/VLM包含解释性文本，以允许用户理解模型关注或识别的内容。 此外，实现允许在人类指导下进行执行前验证和修改的更高层可以促进在这种指导下工作的系统的操作（图4）。

There is a rich history of leveraging reinforcement learning (RL) to train interactive agents that exhibits intelligent behaviors. RL is a methodology to learn the optimal relationship between states and actions based on rewards (or penalties) received as a result of its actions. RL is a highly scalable framework that has been applied to numerous applications including robotics, however, it generally faces several leader-board and LLM/VLMs have shown their potential to mitigate or overcome some of those difficulties:
利用强化学习（RL）来训练具有智能行为的交互式智能体有着丰富的历史。RL是一种学习状态和行为之间的最佳关系的方法，该关系基于作为其行为的结果而收到的奖励（或惩罚）。RL是一种高度可扩展的框架，已应用于包括机器人在内的众多应用，然而，它通常面临几个排行榜，LLM/VLM已经显示出缓解或克服其中一些困难的潜力：

• •

  Reward designing The efficiency of policy learning greatly depends on the design of the reward function. Designing the reward function requires not only knowledge of RL algorithms but also a deep understanding of the nature of the task, and thus often necessitates crafting the function based on expert experience. Several studies explored the use of LLM/VLMs for designing reward functions Yu et al. (2023a); Katara et al. (2023); Ma et al. (2023).
  策略学习的效率在很大程度上取决于奖励函数的设计。设计奖励函数不仅需要RL算法的知识，还需要对任务性质的深入理解，因此通常需要基于专家经验来制作函数。几项研究探索了使用LLM/VLM设计奖励函数Yu et al.（2023 a）; Katara et al.（2023）; Ma et al.（2023）。

• •

  Data collection and efficiency Given its exploratory nature, RL-based policy learning requires a significant amount of data Padalkar et al. (2023). The necessity for extensive data becomes particularly evident when the policy involves managing long sequences or integrating complex actions. This is because these scenarios demand more nuanced decision-making and learning from a wider range of situations. In recent studies, efforts have been directed towards enhancing data generation to support policy learning Kumar et al. (2023); Du et al. (2023). Additionally, in some studies, these models have been integrated into the reward function to improve policy learning Sontakke et al. (2023). Parallel to these developments, another strand of research has focused on achieving parameter efficiency in learning processes using VLMs Tang et al. (2023); Li et al. (2023d) and LLMs Shi et al. (2023)
  数据收集和效率 鉴于其探索性质，基于RL的政策学习需要大量的数据Padalkar et al.（2023）。当策略涉及管理长序列或集成复杂操作时，对大量数据的必要性变得尤为明显。这是因为这些情景需要更细致的决策，并从更广泛的情况中学习。在最近的研究中，努力的方向是加强数据生成，以支持政策学习Kumar et al.（2023）; Du et al.（2023）。此外，在一些研究中，这些模型已被集成到奖励函数中，以改善策略学习Sontakke et al.（2023）。与这些发展并行，另一系列研究集中在使用VLM Tang et  al.（2023）; Li et  al. （2023d）和LLMsShi 等人（2023）

• •

  Long-horizon steps In relation to the issue of data efficiency, RL becomes more challenging as the length of action sequences increases. This is due to the ambiguity in the relationship between actions and rewards, known as the credit assignment problem, and the increase in the number of states to be explored, necessitating a significant amount of time and data. One typical approach for long and complex tasks is to break them down into a sequence of subgoals and apply pretrained policies to solve each subgoal (e.g., Takamatsu et al. (2022)). This idea falls within the framework called the task and motion planning (TAMP)Garrett et al. (2021). TAMP is composed of two primary components: task planning, which entails identifying sequences of high-level actions, and motion planning, which involves finding physically consistent, collision-free trajectories to achieve the objectives of the task plan.
  长水平阶跃 关于数据效率的问题，随着动作序列长度的增加，强化学习变得更具挑战性。这是由于行动和回报之间的关系模糊，即所谓的学分分配问题，以及需要探索的状态数量的增加，需要大量的时间和数据。对于长时间且复杂的任务的一种典型方法是将它们分解成一系列子目标，并应用预先训练的策略来解决每个子目标（例如，Takamatsu等人（2022））。这个想法属于福尔斯的框架称为任务和运动规划（TAMP）加勒特等人。（2021）。TAMP由两个主要部分组成：任务规划，它需要识别高级别动作的序列，以及运动规划，它涉及到找到物理上一致的，无碰撞的轨迹，以实现任务计划的目标。

  LLMs are well-suited to TAMP, and recent research has often adopted an approach where LLMs are used to execute high-level task planning, while low-level controls are addressed with RL-based policies Xu et al. (2023); Sun et al. (2023a); Li et al. (2023b); Parakh et al. (2023). The advanced capabilities of LLMs enable them to effectively decompose even abstract instructions into subgoals Wake et al. (2023c), contributing to the enhancement of language understanding abilities in robotic systems.
  LLMs非常适合TAMP，最近的研究经常采用一种方法，其中LLMs用于执行高级任务规划，而低级控制则通过基于RL的策略来解决Xu et al.（2023）; Sun et al.（2023 a）; Li et al.（2023 b）; Parakh et al.（2023）。LLMs的高级功能使它们能够有效地将抽象指令分解为子目标Wake et al.（2023 c），有助于增强机器人系统的语言理解能力。

While RL aims to train a policy based on exploratory behavior and maximizing rewards through interactions with the environment, imitation learning (IL) seeks to leverage expert data to mimic the actions of experienced agents or experts. For example, in robotics, one of the major frameworks based on IL is Behavioral Cloning (BC). BC is an approach where a robot is trained to mimic the actions of an expert by directly copying them. In this approach, the expert’s actions in performing specific tasks are recorded, and the robot is trained to replicate these actions in similar situations. Recent BC-based methods often incorporate technologies from LLM/VLMs, enabling more advanced end-to-end models. For example, Brohan et al. proposed RT-1 Brohan et al. (2022) and RT-2 Brohan et al. (2023), transformer-based models that output an action sequence for the base and arm, taking a series of images and language as input. These models are reported to show high generalization performance as the result of training on a large amount of training data.
强化学习旨在训练基于探索行为的策略，并通过与环境的交互来最大化奖励，而模仿学习（IL）则试图利用专家数据来模仿有经验的代理或专家的行为。例如，在机器人技术中，基于IL的主要框架之一是行为克隆（BC）。BC是一种训练机器人通过直接模仿专家的动作来模仿他们的方法。在这种方法中，专家在执行特定任务时的动作被记录下来，机器人被训练成在类似的情况下重复这些动作。最近的基于BC的方法通常结合了LLM/VLM的技术，从而实现了更先进的端到端模型。例如，Brohan等人提出了RT-1Brohan等人（2022）和RT-2Brohan等人（2023），基于transformer的模型输出基座和手臂的动作序列，将一系列图像和语言作为输入。 据报道，这些模型在大量训练数据的训练下表现出很高的泛化性能。

Learning intelligent agent behavior leveraging image inputs has been of interest for many years Mnih et al. (2015). The inherent challenge of using RGB input is the curse of dimensionality. To solve this problem, researchers either use more data Jang et al. (2022); Ha et al. (2023) or introduce inductive biases into the model design to improve sample efficiency. In particular, authors incorporate 3D structures into the model architecture for manipulations Zeng et al. (2021); Shridhar et al. (2023); Goyal et al. (2023); James and Davison (2022). For robot navigation, authors Chaplot et al. (2020a, b) leverage maps as a representation. Maps can either be learned from a neural network aggregating all previous RGB inputs or through 3D reconstruction methods such as Neural Radiance Fields Rosinol et al. (2022).
利用图像输入学习智能代理行为多年来一直受到人们的关注，Mnih et al.（2015）。使用RGB输入的固有挑战是维度的灾难。为了解决这个问题，研究人员要么使用更多的数据Jang et al.（2022）; Ha et al.（2023），要么在模型设计中引入归纳偏差以提高样本效率。特别是，作者将3D结构纳入模型架构中进行操作Zeng et al.（2021）; Shridhar et al.（2023）; Goyal et al.（2023）; James and Davison（2022）。对于机器人导航，作者Chaplot等人（2020 a，B）利用地图作为表示。 地图可以从神经网络聚合所有先前的RGB输入或通过3D重建方法（如Neural Radiance FieldsRosinol et al.（2022））学习。

To obtain more data, researchers synthesize synthetic data using graphics simulators Mu et al. (2021); Gong et al. (2023b), and try to close the sim2real gap Tobin et al. (2017); Sadeghi and Levine (2016); Peng et al. (2018). Recently, there has been some collective effort to curate large-scale dataset that aims to resolve the data scarcity problem Padalkar et al. (2023); Brohan et al. (2023). On the other hand, to improve sample complexity, data augmentation techniques have been extensively studied as well Zeng et al. (2021); Rao et al. (2020); Haarnoja et al. (2023); Lifshitz et al. (2023).
为了获得更多数据，研究人员使用图形模拟器Mu et al.（2021）合成合成数据; Gong et al.（2023 b），并试图缩小sim 2 real差距Tobin et al.（2017）; Sadeghi and Levine（2016）; Peng et al.（2018）。最近，已经有一些集体努力来管理大规模数据集，旨在解决数据稀缺问题Padalkar et al.（2023）; Brohan et al.（2023）。另一方面，为了提高样本复杂性，数据增强技术也得到了广泛的研究，Zeng et al.（2021）; Rao et al.（2020）; Haarnoja et al.（2023）; Lifshitz et al.（2023）。

In-context learning was shown to be an effective method for solving tasks in NLP with the advent of large language models like GPT-3 Brown et al. (2020); Min et al. (2022). Few-shot prompts were seen to be an effective way to contextualize model output’s across a variety of tasks in NLP by providing examples of the task within the context of the LLM prompt. Factors like the diversity of examples and quality of examples shown for the in-context demonstrations may improve the quality of model outputs An et al. (2023); Dong et al. (2022). Within the context of multi-modal foundation models, models like Flamingo and BLIP-2 Alayrac et al. (2022); Li et al. (2023c) have been shown to be effective at a variety of visual understanding tasks when given only given a small number of examples. In context learning can be further improved for agents within environments by incorporating environment-specific feedback when certain actions are taken Gong et al. (2023a).
随着大型语言模型的出现，如GPT-3Brown et al.（2020）; Min et al.（2022），上下文学习被证明是解决NLP任务的有效方法。通过在LLM提示符的上下文中提供任务的示例，少镜头提示符被认为是一种有效的方法，可以在NLP中的各种任务中将模型输出上下文化。为上下文演示显示的示例的多样性和示例的质量等因素可能会提高模型输出的质量An et al.（2023）; Dong et al.（2022）。在多模态基础模型的背景下，像Flamingo和BLIP-2这样的模型Alayrac et al.（2022）; Li et al.（2023 c）已经被证明在各种视觉理解任务中是有效的，只要给出少量的例子。 通过在采取某些行动时纳入环境特定的反馈，可以进一步改善环境中的智能体的上下文学习（Gong et al.，2023 a）。

The optimization of agent systems can be divided into spatial and temporal aspects. Spatial optimization considers how agents operate within a physical space to execute tasks. This includes inter-robot coordination, resource allocation, and keeping an organized space.
Agent系统的优化可以分为空间和时间两个方面。空间优化考虑了代理如何在物理空间内操作以执行任务。这包括机器人之间的协调、资源分配和保持有组织的空间。

In order to effectively optimize agent AI systems, especially systems with large numbers of agents acting in parallel, previous works have focused on using large batch reinforcement learning Shacklett et al. (2023). Since datasets of multi-agent interactions for specific tasks are rare, self-play reinforcement learning enables a team of agents to improve over time. However, this may also lead to very brittle agents that can only work under self-play and not with humans or other independent agents since they over-fit to the self-play training paradigm. To address this issue, we can instead discover a diverse set of conventions Cui et al. (2023); Sarkar et al. (2023), and train an agent that is aware of a wide range of conventions. Foundation models can further help to establish conventions with humans or other independent agents, enabling smooth coordination with new agents.
为了有效地优化智能体AI系统，特别是具有大量并行代理的系统，以前的工作主要集中在使用大批量强化学习Shacklett et al.（2023）。由于特定任务的多智能体交互数据集很少，自我强化学习使一组智能体能够随着时间的推移而改进。然而，这也可能导致非常脆弱的代理人，他们只能在自我游戏下工作，而不能与人类或其他独立的代理人一起工作，因为他们过于适合自我游戏训练范例。为了解决这个问题，我们可以发现一组不同的约定Cui et al.（2023）; Sarkar et al.（2023），并训练一个知道各种约定的智能体。基础模型可以进一步帮助建立与人类或其他独立代理的约定，从而实现与新代理的平滑协调。

Temporal optimization, on the other hand, focuses on how agents execute tasks over time. This encompasses task scheduling, sequencing, and timeline efficiency. For instance, optimizing the trajectory of a robot’s arm is an example of efficiently optimizing movement between consecutive tasks Zhou et al. (2023c). At the level of task scheduling, methods like LLM-DP Dagan et al. (2023) and ReAct Yao et al. (2023a) have been proposed to solve efficient task planning by incorporating environmental factors interactively.
另一方面，时间优化关注代理如何随时间推移执行任务。这包括任务调度、排序和时间轴效率。例如，优化机器人手臂的轨迹是有效优化连续任务之间移动的一个例子。 在任务调度层面，已经提出了像LLM-DPDagan et al.（2023）和ReActYao et al.（2023 a）这样的方法，通过交互地结合环境因素来解决有效的任务规划。

Our foray into the agent paradigm involves the development of Agent AI "Modules" for interactive multi-modal agents using LLMs or VLMs. Our initial Agent Modules facilitate training or in-context learning and adopt a minimalist design for the purposes of demonstrating the agent’s ability to schedule and coordinate effectively. We also explored initial prompt-based memory techniques that facilitate better planning and inform future actions approaches within the domain. To illustrate, our “MindAgent" infrastructure comprises 5 main modules: 1) environment perception with task planning, 2) agent learning, 3) memory, 4) general agent action prediction and 5) cognition, as shown in Figure 5.
我们对代理模式的尝试涉及使用LLMs或VLM为交互式多模态代理开发代理AI“模块”。我们的初始代理模块有助于培训或情境学习，并采用了最低限度的设计，以展示代理的能力，以安排和协调有效。我们还探索了最初的基于记忆的技术，这些技术有助于更好地规划和告知该领域内的未来行动方法。为了说明，我们的“MindAgent”基础设施包括5个主要模块：1）环境感知与任务规划，2）代理学习，3）记忆，4）一般代理动作预测和5）认知，如图5所示。

Agent-based AI is a large and fast-growing community within the domains of entertainment, research, and industry. The development of large foundation models has significantly improved the performance of agent AI systems. However, creating agents in this vein is limited by the increasing effort necessary to create high-quality datasets and overall cost. At Microsoft, building high-quality agent infrastructure has significantly impacted multi-modal agent copilots by using advanced hardware, diverse data sources, and powerful software libraries. As Microsoft continues to push the boundaries of agent technology, AI agent platforms are poised to remain a dominant force in the world of multimodal intelligence for years to come. Nevertheless, agent AI interaction is currently still a complex process that requires a combination of multiple skills. The recent advancements in the space of large generative AI models have the potential to greatly reduce the current high cost and time required for interactive content, both for large studios, as well as empowering smaller independent content creators to design high quality experiences beyond what they are currently capable of. The current human-machine interaction systems inside multi-modal agents are primarily rule-based. They do have intelligent behaviors in response to human/user actions and possess web knowledge to some extent. However, these interactions are often limited by software development costs to enable specific behaviors in the system. In addition, current models are not designed to help human to achieve a goal in the case of users’ inability to achieve specific tasks. Therefore, there is a need for an agent AI system infrastructure to analyze users behaviors and provide proper support when needed.
基于Agent的人工智能是一个大型的、快速发展的社区，在娱乐、研究和工业领域都有广泛的应用。大型基础模型的发展显著提高了智能体人工智能系统的性能。然而，以这种方式创建代理受到创建高质量数据集所需的不断增加的工作量和总体成本的限制。在Microsoft，构建高质量的代理基础架构通过使用先进的硬件、多样化的数据源和功能强大的软件库，对多模式代理副本产生了重大影响。随着微软不断推进代理技术的边界，AI代理平台将在未来几年保持多模式智能世界的主导力量。尽管如此，智能体AI交互目前仍然是一个复杂的过程，需要多种技能的组合。 大型生成式人工智能模型领域的最新进展有可能大幅降低大型工作室当前交互式内容所需的高成本和时间，并使规模较小的独立内容创作者能够设计出超出其当前能力的高质量体验。目前多模态智能体内部的人机交互系统主要是基于规则的。它们确实具有响应人类/用户动作的智能行为，并在一定程度上拥有网络知识。然而，这些交互通常受到软件开发成本的限制，以实现系统中的特定行为。此外，当前的模型并不是设计成在用户不能实现特定任务的情况下帮助人类实现目标。因此，需要一个代理人工智能系统基础设施来分析用户行为，并在需要时提供适当的支持。

Action agents refer to the agents that need to execute physical actions in the simulated physical environment or real world. In particular, they need to be actively engaging in activities with the environment. We broadly classify action agents into two different categories based on their application domains: gaming AI and robotics.
动作智能体是指需要在模拟的物理环境或真实的世界中执行物理动作的智能体。特别是，他们需要积极参与与环境有关的活动。我们根据其应用领域将动作代理大致分为两类：游戏AI和机器人。

In gaming AI, the agents will interact with the game environment and other independent entities. In these settings, natural language can enable smooth communication between agents and humans. Depending on the game, there may be a specific task to accomplish, providing a true reward signal. For instance, in the competitive Diplomacy game, training a language model using human conversation data along with an action policy with RL enables human-level play Meta Fundamental AI Research Diplomacy Team et al.(2022)Meta Fundamental AI Research (FAIR) Diplomacy Team, Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al. (FAIR).
在游戏AI中，智能体将与游戏环境和其他独立实体进行交互。在这些环境中，自然语言可以实现智能体和人类之间的顺畅沟通。根据游戏的不同，可能会有一个特定的任务要完成，提供一个真正的奖励信号。例如，在竞争性的外交游戏中，使用人类对话数据沿着RL的行动策略训练语言模型可以实现人类水平的游戏 。（2022）Meta基础人工智能研究（FAIR）外交团队，巴赫金，布朗，迪南，法里纳，弗莱厄蒂，弗里德，戈夫，格雷，胡 等。（FAIR）。

There are also settings where we agents act as normal residents in a town Park et al. (2023a), without trying to optimize a specific goal. Foundation models are useful in these settings because they can model interactions that appear more natural by mimicking human behavior. When augmented with external memory, they produce convincing agents that can have conversations, daily schedules, form relationships, and have a virtual life.
也有一些设置，我们代理人作为城镇中的正常居民Park et al.（2023a），而不试图优化特定的目标。基础模型在这些设置中非常有用，因为它们可以通过模拟人类行为来模拟看起来更自然的交互。当外部记忆被增强时，他们会产生令人信服的代理人，这些代理人可以进行对话、安排日程、建立关系，并拥有虚拟生活。

Interactive agents simply refer to agents that can interact with the world, a broader class of agents than action agents. Their forms of interaction do not necessarily require physical actions, but may involve communicating information to users or modifying the environment. For instance, an embodied interactive agent may answer a user’s questions about a topic through dialogue or help users parse through existing information similar to a chatbot. By extending an agent’s capabilities to include information sharing, the core designs and algorithms of Agent AI can be effectively adapted for a range of applications, such as diagnostic Lee et al. (2023) and knowledge-retrieval Peng et al. (2023) agents.
交互式代理简单地指可以与世界交互的代理，这是一个比行动代理更广泛的代理类别。它们的交互形式不一定需要物理动作，但可能涉及向用户传达信息或修改环境。例如，具体化的交互式代理可以通过对话回答用户关于主题的问题，或者类似于聊天机器人帮助用户解析现有信息。通过扩展Agent的功能以包括信息共享，Agent AI的核心设计和算法可以有效地适用于一系列应用，例如诊断Lee et al.（2023）和知识检索Peng et al.（2023）Agent。

AR/VR/mixed-reality (jointly referred to as XR) settings currently require skilled artists and animators to create characters, environments, and objects to be used to model interactions in virtual worlds. This is a costly process that involves concept art, 3D modeling, texturing, rigging, and animation. XR agents can assist in this process by facilitating interactions between creators and building tools to help build the final virtual environment.
AR/VR/混合现实（统称为XR）设置目前需要熟练的艺术家和动画师创建角色，环境和对象，用于模拟虚拟世界中的交互。这是一个昂贵的过程，涉及概念艺术，3D建模，纹理，索具和动画。XR代理可以通过促进创建者和构建工具之间的交互来帮助构建最终的虚拟环境，从而在此过程中提供帮助。

Our early experiments have already demonstrated that GPT models can be used in the few-shot regime inside of the Unity engine (without any additional fine-tuning) to call engine-specific methods, use API calls to download 3d models from the internet and place them into the scene, and assign state trees of behavior and animations to them Huang et al. (2023a). This behavior likely emerges due to the presence of similar code in open source game repositories that use Unity. Therefore, GPT models are capable of building rich visual scenes in terms of loading in many objects into the scene from a simple user prompt.
我们早期的实验已经证明，GPT模型可以在Unity引擎内部的几个镜头中使用（无需任何额外的微调），以调用引擎特定的方法，使用API调用从互联网下载3D模型并将其放置到场景中，并为它们分配行为和动画的状态树Huang et al.（2023 a）。这种行为可能是由于使用Unity的开源游戏存储库中存在类似的代码而出现的。因此，GPT模型能够通过从简单的用户提示将许多对象加载到场景中来构建丰富的视觉场景。

The aim of this category of agents is to build a platform and a set of tools that provide an efficient interface between large AI models (both GPT-family ones as well as diffusion image models) and a rendering engine. We explore two primary avenues here:
这类代理的目的是构建一个平台和一组工具，在大型AI模型（GPT系列模型和扩散图像模型）和渲染引擎之间提供有效的接口。我们在这里探索两个主要途径：

• •

  Integration of large models into the various editor tools in the agent infrastructure, allowing for significant speedups in development.
  将大型模型集成到代理基础设施中的各种编辑器工具中，从而大大加快了开发速度。

• •

  Controlling the rendering engine from within a user experience, by generating code that follows user instruction and then compiling it at runtime, allowing for users to potentially edit the VR/simulation they are interacting with in arbitrary ways, even by introducing new agent mechanics.
  从用户体验中控制渲染引擎，通过生成遵循用户指令的代码，然后在运行时编译它，允许用户以任意方式编辑他们正在交互的VR/模拟，甚至通过引入新的代理机制。

Introducing an AI copilot focused on XR settings would be useful for XR creators, who can use the copilot to complete tedious tasks, like providing simple assets or writing code boilerplate, freeing creators to focus on their creative vision and quickly iterate on ideas.
引入一个专注于XR设置的人工智能副驾驶仪对XR创作者来说是有用的，他们可以使用副驾驶仪完成繁琐的任务，比如提供简单的资产或编写代码样板，让创作者专注于他们的创意愿景并快速实现想法。

Furthermore, agents can help users interactively modify the environment by adding new assets, changing the dynamics of the environment, or building new settings. This form of dynamic generation during runtime can also be specified by a creator, enabling the user’s experience to feel fresh and continue evolving over time.
此外，代理可以帮助用户交互式地修改环境，通过添加新的资产，改变环境的动态，或建立新的设置。这种在运行时动态生成的形式也可以由创建者指定，从而使用户的体验能够感到新鲜并随着时间的推移而继续发展。

Knowledge Agents reason over their acquired knowledge systems in two directions: implicit and explicit. Implicit knowledge is typically what large-scale language models like the GPT series Brown et al. (2020); OpenAI (2023) encapsulate after being trained on vast amounts of text data. These models can generate responses that give the impression of understanding, as they draw on patterns and information implicitly learned during training. Explicit knowledge, conversely, is structured and can be directly queried, such as the information found in knowledge bases or databases, which was traditionally used to enhance AI reasoning capabilities by referencing verifiable external resources.
知识主体在两个方向上对他们获得的知识系统进行推理：隐式和显式。隐式知识通常是大规模语言模型，如GPT系列Brown et al.（2020）; OpenAI（2023）在大量文本数据上训练后封装的。这些模型可以产生给人理解的印象的反应，因为它们利用了在培训期间隐性学习的模式和信息。相反，显性知识是结构化的，可以直接查询，比如在知识库或数据库中找到的信息，传统上是通过引用可验证的外部资源来增强AI推理能力。

Despite the advancements in language models, their implicit knowledge is static and becomes outdated as the world evolves Lewis et al. (2020); Peng et al. (2023). This limitation necessitates the integration of explicit knowledge sources that are updated continuously, ensuring that AI systems can provide accurate and current responses. The fusion of implicit and explicit knowledge equips AI agents with a more nuanced understanding and the ability to apply knowledge contextually, akin to human intelligence Gao et al. (2022). Such integration is crucial for crafting knowledge-centric AI agents that not only possess information but can also understand, explain, and employ it, thereby narrowing the chasm between extensive learning and profound knowledge (Marcus and Davis, 2019; Gao et al., 2020). These agents are designed to reason with flexibility and dynamic information about the world, enhancing their robustness and adaptability (Marcus, 2020).
尽管语言模型取得了进步，但它们的隐性知识是静态的，随着世界的发展而变得过时刘易斯等人（2020）;彭等人（2023）。这一限制需要整合不断更新的显性知识源，以确保人工智能系统能够提供准确和最新的响应。隐式和显式知识的融合使AI代理具有更细致的理解和根据上下文应用知识的能力，类似于人类智能Gao et al.（2022）。这种整合对于制作以知识为中心的人工智能代理至关重要，这些代理不仅拥有信息，而且还可以理解，解释和使用它，从而缩小广泛学习和深刻知识之间的鸿沟（Marcus和Davis，2019; Gao 等人，2020年）。 这些智能体被设计成能够灵活地推理和动态地了解世界，增强它们的鲁棒性和适应性（Marcus，2020）。

Generally, a logic agent is a component of a system designed to apply logical reasoning to process data or solve tasks specific to logical inference or logical reasoning. Logic agents within the context of large foundation models like GPT-4 refers to a specialized component or submodules designed to handle logical reasoning tasks. These tasks often involve understanding and manipulating abstract concepts, deducing conclusions from given premises, or solving problems that require a structured, logical approach. Broadly, foundation models like GPT-4 are trained on a vast corpus of text data and learn to perform a wide range of tasks, including those that require some form of logical reasoning. Thus, their capability for logical reasoning is integrated into the overall architecture, and they generally do not possess a distinct, isolated "Logic agent". While GPT-4 and similar models can perform tasks that involve logic, their approach is fundamentally different from how humans or traditional logic-based systems operate. They do not follow formal logical rules or have an explicit understanding of logic; rather, they generate responses based on patterns learned from the training data. As a result, their performance in logical tasks can be impressive, but it can also be inconsistent or limited by the nature of the training data and the inherent limitations of the model’s design. One example of embedding a separate logical submodule into the architecture is (Wang et al., 2023e), which modifies the token embedding process used by LLMs during pre-training by parsing text into logical segments and explicitly modeling logical hierarchies in the token embeddings.
通常，逻辑代理是被设计为应用逻辑推理来处理数据或解决特定于逻辑推理或逻辑推理的任务的系统的组件。在大型基础模型（如GPT-4）的上下文中，逻辑代理是指专门用于处理逻辑推理任务的组件或子模块。这些任务通常涉及理解和操作抽象概念，从给定的前提推导结论，或解决需要结构化，逻辑方法的问题。广义地说，像GPT-4这样的基础模型是在大量文本数据的语料库上训练的，并学习执行广泛的任务，包括那些需要某种形式的逻辑推理的任务。因此，它们的逻辑推理能力被集成到整个体系结构中，并且它们通常不拥有独特的、孤立的“逻辑代理”。 虽然GPT-4和类似的模型可以执行涉及逻辑的任务，但它们的方法与人类或传统的基于逻辑的系统的操作方式有着根本的不同。它们不遵循正式的逻辑规则或对逻辑有明确的理解;相反，它们基于从训练数据中学习到的模式生成响应。因此，它们在逻辑任务中的性能可能令人印象深刻，但也可能不一致或受到训练数据的性质和模型设计的固有限制的限制。将单独的逻辑子模块嵌入到体系结构中的一个示例是（Wang 等人，2023 e），其通过将文本解析成逻辑段并显式地对令牌嵌入中的逻辑层次结构建模来修改由LLMs在预训练期间使用的令牌嵌入过程。

Emotional understanding and empathy are important skills for agents in many human-machine interactions. To illustrate, one important goal for creating engaging dialogue agents is to have the agents act with increased emotion and empathy while minimizing socially inappropriate or offensive outputs. To advance towards this goal for dialogue agents, we released the Neural Image Commenting with Empathy (NICE) dataset Chen et al. (2021) consisting of almost two million images and the corresponding human-generated comments and a set of human emotion annotations. We also provided a novel pre-training model - Modeling Affect Gneration for Image Comments (MAGIC) Chen et al. (2021) - which aims to generate comments for images, conditioned on linguistic representations that capture style and affect, and to help generate more empathetic, emotional, engaging and socially appropriate comments. Our experiments show that the approach is effective in training a more human-like and engaging image comment agent. Developing empathy-aware agents is a promising direction for interactive agents, and it is important to create agents with emotional understanding capabilities across a wide range of groups and populations, especially considering that many current language models exhibit bias in their emotional understanding and empathetic reasoning capabilities Mao et al. (2022); Wake et al. (2023d).
情感理解和同理心是智能体在许多人机交互中的重要技能。为了说明，创建参与对话代理的一个重要目标是让代理以增加的情感和同理心行事，同时最大限度地减少社交不适当或冒犯性的输出。为了实现对话代理的这一目标，我们发布了Chen et al.（2021）的Neural Image Commenting with Empathy（NICE）数据集，其中包括近200万张图像和相应的人类生成的评论以及一组人类情感注释。我们还提供了一种新的预训练模型-为图像评论建模情感生成（MAGIC）Chen et al.（2021）-旨在为图像生成评论，以捕捉风格和影响的语言表示为条件，并帮助生成更具同情心，情感，吸引力和社会适当的评论。 实验结果表明，该方法能够有效地训练出更人性化、更吸引人的图像评论Agent。开发移情感知代理是交互式代理的一个有前途的方向，重要的是要创建具有跨广泛群体和人群的情感理解能力的代理，特别是考虑到许多当前的语言模型在情感理解和移情推理能力方面表现出偏见Mao et al.（2022）; Wake et al.（2023 d）。

Neuro-Symbolic agents operate on a hybrid system of neurons and symbols d’Avila Garcez and Lamb (2020). To solve problems stated in natural language is a challenging task because it requires explicitly capturing discrete symbolic structural information implicit in the input. However, most general neural sequence models do not explicitly capture such structural information, limiting their performance on these tasks. The work Chen et al. (2020) propose a new encoder-decoder model based on a structured neural representation agent, The encoder of TP-N2F employs TPR ‘binding’ to encode natural-language symbolic structure in vector space and the decoder uses TPR ‘unbinding’ to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments.
神经符号代理在神经元和符号的混合系统上运行d 'Avila Garcez和Lamb（2020）。解决自然语言表述的问题是一项具有挑战性的任务，因为它需要显式地捕获隐含在输入中的离散符号结构信息。然而，大多数一般的神经序列模型并没有明确地捕获这样的结构信息，限制了它们在这些任务上的性能。Chen et al.（2020）提出了一种基于结构化神经表示代理的新的编码器-解码器模型，TP-N2 F的编码器采用TPR“绑定”来编码向量空间中的自然语言符号结构，解码器使用TPR“解绑定”来生成符号空间中由关系元组表示的顺序程序，每个元组由关系（或操作）和许多参数组成。

Instruction following vision-language (VL) models like GPT-4 offer a flexible interface that supports a broad range of multimodal tasks in a zero-shot fashion. However, interfaces that operate on full images do not directly enable the user to “point to” and access specific regions within images. This capability is important not only to support reference-grounded VL benchmarks, but also, for practical applications that require precise within-image reasoning. In (Park et al., 2023b), we build Localized Visual Commonsense model which allows users to specify (multiple) regions-as-input. We train our model by sampling localized commonsense knowledge from a large language model (LLM): specifically, we prompt a LLM to collect common sense knowledge given a global literal image description and a local literal region description automatically generated by a set of VL models. This pipeline is scalable and fully automatic, as no aligned or human-authored image and text pairs are required. With a separately trained critic model that selects high quality examples, we find that training on the localized commonsense corpus expanded solely from images can successfully distill existing VL models to support a reference-as-input interface. Empirical results and human evaluations in zero-shot settings demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression.
像GPT-4这样的指令跟随视觉语言（VL）模型提供了一个灵活的界面，以零触发的方式支持广泛的多模态任务。然而，在完整图像上操作的界面不能直接使用户“指向”和访问图像内的特定区域。这种能力不仅对于支持基于参考的VL基准测试非常重要，而且对于需要精确图像内推理的实际应用也非常重要。在（Park 等人，2023 b），我们构建了本地化视觉常识模型，该模型允许用户指定（多个）区域作为输入。我们通过从大型语言模型（LLM）中采样本地化常识知识来训练我们的模型：具体地说，我们提示LLM收集常识知识，给出一组VL模型自动生成的全局文字图像描述和局部文字区域描述。 此管道是可伸缩的，并且是全自动的，因为不需要对齐或人工创作的图像和文本对。通过单独训练的评论家模型选择高质量的示例，我们发现仅从图像扩展的本地化常识语料库上的训练可以成功地提取现有的VL模型以支持参考作为输入接口。实验结果和在零触发设置下的人类评估表明，与传递生成的指称表达式的基线相比，我们的提取方法导致更精确的推理VL模型。

In modern gaming systems, the behavior of Non-Player Characters (NPCs) is predominantly dictated by predefined scripts crafted by developers. These scripts encompass a range of reactions and interactions based on various triggers or player actions within the gaming environment. However, this scripted nature often results in predictable or repetitive NPC behavior which fails to evolve in response to player’s actions or the dynamic environment of the game. This rigidity hampers the immersive experience intended in a dynamic gaming environment. Therefore, there is a burgeoning interest in leveraging LLMs to induce autonomy and adaptability in NPC behavior, making interactions more nuanced and engaging. AI-driven NPCs can learn from player behavior, adapt to varying strategies, and provide a more challenging and less predictable gameplay experience. Large Language Models (LLMs) can significantly contribute to evolving NPC behavior in games. By processing vast amounts of text, LLMs can learn patterns and generate responses that are more varied and human-like. They can be utilized to create dynamic dialogue systems, making interactions with NPCs more engaging and less predictable. Furthermore, LLMs can be trained on player feedback and in-game data to continually refine NPC behaviors, making them more attuned to player expectations and game dynamics.
在现代游戏系统中，非玩家角色（NPC）的行为主要由开发人员精心制作的预定义脚本决定。这些脚本包含基于游戏环境中的各种触发器或玩家动作的一系列反应和交互。然而，这种脚本化的性质通常会导致可预测或重复的NPC行为，这些行为无法响应玩家的动作或游戏的动态环境。这种刚性阻碍了动态游戏环境中的沉浸式体验。因此，人们越来越有兴趣利用LLMs来诱导NPC行为的自主性和适应性，使互动更加细致入微和引人入胜。AI驱动的NPC可以从玩家行为中学习，适应不同的策略，并提供更具挑战性和更少可预测性的游戏体验。大型语言模型（LLMs）可以显著促进游戏中NPC行为的发展。 通过处理大量的文本，LLMs可以学习模式并生成更多样化和更人性化的响应。它们可以用来创建动态对话系统，使与NPC的互动更具吸引力，更不可预测。此外，LLMs可以根据玩家反馈和游戏中的数据进行训练，以不断改进NPC的行为，使其更加适应玩家的期望和游戏动态。

The interaction between human players and NPCs is a crucial aspect of the gaming experience. The conventional interaction paradigm is primarily one-dimensional, with NPCs reacting in a preset manner to player inputs. This limitation stifles the potential for a more organic and enriching interaction, akin to human-human interaction within the virtual realm. The advent of LLM and VLM technologies holds the promise of transforming this paradigm. By employing these technologies, gaming systems can analyze and learn from human behavior to provide more human-like interactions. This not only enhances the realism and engagement of the game but also provides a platform for exploring and understanding human-machine interaction in a controlled yet complex setting.
人类玩家和NPC之间的互动是游戏体验的一个重要方面。传统的互动模式主要是一维的，NPC以预设的方式对玩家的输入做出反应。这种限制扼杀了更有机和丰富的互动的潜力，类似于虚拟领域中的人与人的互动。LLM和VLM技术的出现有望改变这种模式。通过采用这些技术，游戏系统可以分析和学习人类行为，以提供更人性化的交互。这不仅增强了游戏的真实感和参与度，还为在受控但复杂的环境中探索和理解人机交互提供了平台。

Gaming is an integral part of daily life, estimated to engage half of the world’s population³³3https://www.dfcint.com/global-video-game-audience-reaches-3-7-billion/. Additionally, it exhibits a positive impact on mental health⁴⁴4https://news.microsoft.com/source/features/work-life/mind-games-how-gaming-can-play-a-positive-role-in-mental-health/. However, contemporary game systems exhibit a deficiency in interactions with human players since their behaviors are primarily hand-crafted by game developers. These pre-programmed behaviors frequently fail to adapt to players’ needs. Consequently, there exists a need for new AI systems in games that can analyze player behaviors and furnish appropriate support when necessary. Intelligent interactive systems bear the potential to revolutionize how gamers interact with gaming systems in general. NPCs’ interactions with gamers are no longer confined by the restricted rule sets designed by game developers. They have the potential to adapt seamlessly to gamers’ experiences, providing timely feedback to enrich the gaming experience and elevate the synergy of human-machine interaction.
游戏是日常生活中不可或缺的一部分，估计吸引了世界一半的人口 ³³3https://www.dfcint.com/global-video-game-audience-reaches-3-7-billion/ 。此外，它对心理健康也有积极的影响#1。然而，当代游戏系统在与人类玩家的交互方面表现出不足，因为他们的行为主要是由游戏开发者手工制作的。这些预先设定好的行为经常不能适应玩家的需要。因此，游戏中需要新的AI系统，可以分析玩家行为并在必要时提供适当的支持。智能交互系统有可能彻底改变玩家与游戏系统的交互方式。NPC与玩家的互动不再受游戏开发者设计的限制性规则集的限制。它们有潜力无缝适应玩家的体验，提供及时的反馈，以丰富游戏体验并提升人机交互的协同作用。

LLMs can serve as a robust tool for analyzing in-game text data, including chat logs, player feedback, and narrative content. They can help in identifying patterns of player behavior, preferences, and interactions which can be invaluable for game developers to improve game mechanics and narratives. Additionally, VLMs can parse through large quantities of image and video data from gaming sessions to help analyze user intent and actions within the game world. Moreover, LLMs and VLMs can facilitate the development of intelligent agents within games that can communicate with players and other agents in a sophisticated and human-like manner, enhancing the overall gaming experience. Beyond LLMs and VLMs, user input data, provides a promising avenue for creating game-playing agents that model perception, game playing, and game understanding by imitating human players. By incorporating a combination of player interactions and feedback, pixel inputs, and natural language planning and understanding, agent models can assist in the continuous improvement of game dynamics, driving a more player-centric evolution of the gaming environment.
LLMs可以作为一个强大的工具，用于分析游戏中的文本数据，包括聊天记录，玩家反馈和叙事内容。它们可以帮助识别玩家行为、偏好和交互的模式，这对游戏开发人员改善游戏机制和叙事是非常宝贵的。此外，VLM可以解析来自游戏会话的大量图像和视频数据，以帮助分析游戏世界中的用户意图和动作。此外，LLMs和VLM可以促进游戏中智能代理的开发，这些智能代理可以以复杂和人性化的方式与玩家和其他代理进行通信，从而增强整体游戏体验。除了LLMs和VLM之外，用户输入数据为创建游戏代理提供了一个有前途的途径，这些代理通过模仿人类玩家来模拟感知，游戏和游戏理解。 通过整合玩家交互和反馈、像素输入以及自然语言规划和理解的组合，智能体模型可以帮助持续改进游戏动态，推动游戏环境以玩家为中心的演变。

Scene synthesis is a vital component in the creation and enhancement of immersive gaming environments. It entails the automatic or semi-automatic generation of three-dimensional (3D) scenes and environments within a game. This process includes the generation of terrain, placement of objects, creation of realistic lighting, and sometimes even dynamic weather systems.
场景合成是创建和增强沉浸式游戏环境的重要组成部分。它需要在游戏中自动或半自动生成三维（3D）场景和环境。此过程包括地形的生成、对象的放置、真实照明的创建，有时甚至包括动态天气系统。

Modern games often feature vast, open-world environments. Manually designing these landscapes can be incredibly time-consuming and resource-intensive. Automated terrain generation, often leveraging procedural or AI-driven techniques, can produce complex, realistic landscapes with less manual effort. LLMs and VLMs can utilize the internet scale knowledge to formulate rules to design non-repeating landscapes that are visually impressive and unique. Additionally, LLMs and VLMs can be used to ensure the semantic consistency and variability of generated assets. Placing objects such as buildings, vegetation, and other elements within a scene in a realistic and aesthetically pleasing manner is crucial for immersion.
现代游戏通常以广阔、开放的世界环境为特色。手动设计这些景观可能非常耗时和资源密集。自动地形生成通常利用程序或人工智能驱动的技术，可以生成复杂、逼真的地形，而人工工作量更少。LLMs和VLM可以利用互联网规模知识来制定规则，以设计视觉上令人印象深刻和独特的非重复景观。此外，LLMs和VLM可用于确保生成的资产的语义一致性和可变性。以逼真且美观的方式在场景中放置对象（如建筑物、植被和其他元素）对于沉浸效果至关重要。

VLMs and LLMs can assist in object placement by adhering to predefined or learned rules and aesthetics, thus speeding up the level design process. VLMs and LLMs can be further trained to understand the principles of design and aesthetics, aiding in the procedural generation of content. They can help formulate rules or guidelines that procedural algorithms can follow to generate objects, and scenes that are both visually appealing and contextually appropriate.
VLM和LLMs可以通过遵守预定义或学习的规则和美学来帮助对象放置，从而加快关卡设计过程。VLM和LLMs可以进一步培训，以了解设计和美学的原则，帮助内容的程序生成。它们可以帮助制定规则或指导方针，程序算法可以遵循这些规则或指导方针来生成视觉吸引力和上下文适当的对象和场景。

Realistic lighting and atmospheric effects are fundamental for creating a believable and engaging gaming environment. Advanced algorithms can simulate natural lighting conditions and dynamic weather effects, enhancing the realism and mood of the scene. LLMs can help develop systems to acheive more realistic lighting and atmospheric effects in several innovative ways. VLMs can analyze vast datasets from real-world lighting and atmospheric conditions to help develop more realistic algorithms for simulating these effects in games. By understanding the patterns and intricacies of natural lighting and weather, these models can contribute to the development of algorithms that mimic reality closely. LLMs and VLMs could also be used to develop systems that adjust lighting and atmospheric effects in real-time based on player actions, game states, or external inputs. They can process natural language commands from players to modify the game environment, providing a more interactive and immersive experience.
逼真的灯光和氛围效果是创建可信和引人入胜的游戏环境的基础。先进的算法可以模拟自然光照条件和动态天气效果，增强场景的真实感和情绪。LLMs可以帮助开发系统，以多种创新方式实现更逼真的照明和大气效果。VLM可以分析来自真实世界照明和大气条件的大量数据集，以帮助开发更逼真的算法来模拟游戏中的这些效果。通过了解自然光和天气的模式和复杂性，这些模型可以为开发紧密模拟现实的算法做出贡献。LLMs和VLM还可以用于开发基于玩家动作、游戏状态或外部输入实时调整照明和氛围效果的系统。 它们可以处理玩家的自然语言命令来修改游戏环境，提供更具交互性和沉浸式的体验。

Recent research has demonstrated the potential of LLM/VLMs for robotic agents that involve interactions with humans in an environment. Research topics that aim to leverage latest LLM/VLM technologies include:
最近的研究已经证明了LLM/VLM对于机器人代理的潜力，这些机器人代理涉及与环境中的人类的交互。旨在利用最新LLM/VLM技术的研究主题包括：

Multimodal Systems: Recent research has been actively focusing on developing end-to-end systems that incorporate the latest LLM and VLM technologies as encoders for input information. Particularly, there is a significant trend towards modifying these foundation models to process multimodal information. Jiang et al. (2022); Brohan et al. (2023, 2022); Li et al. (2023d); Ahn et al. (2022b); Shah et al. (2023b); Li et al. (2023e). This adaptation aims to guide robotic actions based on both linguistic instructions and visual cues, thus achieving an effective embodiment.
多模式系统：最近的研究一直在积极关注开发端到端系统，这些系统将最新的LLM和VLM技术作为输入信息的编码器。特别地，存在修改这些基础模型以处理多模态信息的显著趋势。Jiang等人（2022）; Brohan等人（2023，2022）; Li等人（2023 d）; Ahn等人（2022 b）; Shah等人（2023 b）; Li等人（2023 e）。该适应性修改旨在基于语言指令和视觉线索两者来引导机器人动作，从而实现有效的实施例。

Task Planning and Skill Training: In contrast to end-to-end systems, Task And Motion Planning (TAMP) based systems first compute a high-level task plan and then achieve them with low-level robot control, known as skills.
任务规划和技能培训：与端到端系统相比，基于任务和运动规划（TAMP）的系统首先计算高级任务计划，然后通过低级机器人控制（称为技能）实现这些计划。

The advanced language processing abilities of LLMs have demonstrated the capability to interpret instructions and decompose them into robot action steps, greatly advancing task planning technologies Ni et al. (2023); Li et al. (2023b); Parakh et al. (2023); Wake et al. (2023c). For skill training, several studies have explored the use of LLMs/VLMs for designing reward functions Yu et al. (2023a); Katara et al. (2023); Ma et al. (2023), generating data to facilitate policy learning Kumar et al. (2023); Du et al. (2023), or serving as part of a reward function Sontakke et al. (2023). Together with training frameworks such as RL and IL, these efforts will contribute to the development of efficient robot controllers.
LLMs的高级语言处理能力已经证明了解释指令并将其分解为机器人动作步骤的能力，极大地推进了任务规划技术Ni et al.（2023）; Li et al.（2023 b）; Parakh et al.（2023）; Wake et al.（2023 c）。对于技能培训，一些研究探索了使用LLMs/VLM设计奖励函数Yu et al.（2023 a）; Katara et al.（2023）; Ma et al.（2023），生成数据以促进策略学习Kumar et al.（2023）; Du et al.（2023），或作为奖励函数Sontakke et al.（2023）的一部分。与RL和IL等训练框架一起，这些努力将有助于开发高效的机器人控制器。

On-site Optimization: Executing long task steps in robotics can be difficult due to unexpected and unpredictable environmental conditions. Therefore, a significant challenge in the field of robotics involves dynamically adapting and refining robotic skills by integrating task plans with real-time environmental data. For instance, (Ahn et al., 2022b) proposed an approach that calculates the feasibility of actions (i.e., affordance) from visual information and compares it with planned tasks. Additionally, there are approaches that focus on enabling LLMs to output the pre-conditions and post-conditions (e.g., states of objects and their interrelationships) of task steps to optimize their execution Zhou et al. (2023c) and detect pre-condition errors for necessary revisions to the task plan Raman et al. (2023). These strategies seek to achieve environment-grounded robot execution by integrating environmental information and adjusting the robot’s actions at the task plan or controller level.
现场优化：由于意外和不可预测的环境条件，在机器人中执行长任务步骤可能会很困难。因此，在机器人领域的一个重大挑战涉及通过将任务计划与实时环境数据集成来动态地适应和改进机器人技能。例如，（Ahn 等人，2022b）提出了一种计算动作的可行性（即，示能）并将其与计划的任务进行比较。另外，存在专注于使LLMs能够输出前置条件和后置条件（例如，对象的状态及其相互关系），以优化其执行Zhou et al.（2023 c），并检测任务计划拉曼et al.（2023）的必要修订的先决条件错误。 这些策略通过整合环境信息和在任务计划或控制器级别调整机器人的动作来寻求实现基于环境的机器人执行。

Conversation Agents: In creating conversational robots, LLMs can contribute to natural, context-sensitive interactions with humans Ye et al. (2023a); Wake et al. (2023f). These models process and generate responses that mimic human conversation, allowing robots to participate in meaningful dialogues. Additionally, LLMs play a significant role in the estimation of conceptual Hensel et al. (2023); Teshima et al. (2022) and emotional attributes Zhao et al. (2023); Yang et al. (2023b); Wake et al. (2023d) of utterances. Those attributes facilitate the understanding of human intent and meaningful gesture generation, thus contributing to the naturalness and efficacy of human-robot communication.
会话代理：在创建会话机器人时，LLMs可以促进与人类的自然，上下文敏感的交互Ye et al.（2023 a）; Wake et al.（2023 f）。这些模型处理并生成模仿人类对话的响应，允许机器人参与有意义的对话。此外，LLMs在概念Hensel et al.（2023）; Teshima et al.（2022）和话语的情感属性Zhao et al.（2023）; Yang et al.（2023 b）; Wake et al.（2023 d）的估计中发挥重要作用。这些属性有助于理解人类意图和有意义的手势生成，从而有助于人类与机器人通信的自然性和有效性。

Navigation Agents: Robot navigation has a long history of research, focusing on core aspects such as map-based path planning and Simultaneous Localization and Mapping (SLAM) for creating environmental maps. These functionalities have become standard in widely used robot middleware like the Robot Operating System (ROS) Guimarães et al. (2016).
导航代理： 机器人导航的研究由来已久，主要集中在基于地图的路径规划和用于创建环境地图的同步定位与地图绘制（SLAM）等核心方面。这些功能已成为广泛使用的机器人中间件的标准，如机器人操作系统（ROS）Guimarces et al.（2016）。

While classic navigation techniques remain prevalent in many robotics applications, they typically rely on static or pre-created maps. Recently, there has been an increased interest in advanced technologies that enable robots to navigate in more challenging environments, leveraging breakthroughs in fields like computer vision and natural language processing. One representative task is object navigation Chaplot et al. (2020a); Batra et al. (2020); Gervet et al. (2023); Ramakrishnan et al. (2022); Zhang et al. (2021), where robots use object names for navigation instead of map coordinates, requiring the visual grounding of object names in the environment. Furthermore, recent attention has been given to technologies that navigate robots in entirely unfamiliar new environments on a zero-shot basis, on top of foundation models, so-called zero-shot object navigation Gadre et al. (2023); Dorbala et al. (2023); Cai et al. (2023). Additionally, Vision-Language Navigation (VLN) Anderson et al. (2018a) is a representative task, where the task involves navigating an agent by natural language instructions in previously unseen, real-world environments Shah et al. (2023a); Zhou et al. (2023a); Dorbala et al. (2022); Liang et al. (2023); Huang et al. (2023b). VLN interprets sentences rather than object names, such as “go to the bathroom on your left.,” thus it requires a higher functionality to parse input text Wang et al. (2019).
虽然经典的导航技术在许多机器人应用中仍然很流行，但它们通常依赖于静态或预先创建的地图。最近，人们对先进技术越来越感兴趣，这些技术使机器人能够在更具挑战性的环境中导航，利用计算机视觉和自然语言处理等领域的突破。一个代表性的任务是对象导航Chaplot等人（2020 a）; Batra等人（2020）; Gervet等人（2023）; Ramakrishnan等人（2022）; Zhang等人（2021），其中机器人使用对象名称而不是地图坐标进行导航，需要环境中对象名称的视觉基础。 此外，最近的注意力已经被给予在完全不熟悉的新环境中在零拍摄基础上导航机器人的技术，在基础模型之上，所谓的零拍摄对象导航Gadre et al.（2023）; Dorbala et al.（2023）; Cai et al.（2023）。此外，视觉语言导航（VLN）安德森等人（2018 a）是一个代表性的任务，其中任务涉及在以前看不见的真实世界环境中通过自然语言指令导航智能体Shah等人（2023 a）; Zhou等人（2023 a）; Dorbala等人（2022）; Liang等人（2023）; Huang等人（2023 b）。 VLN解释句子而不是对象名称，例如“去你左边的浴室。”因此，它需要更高的功能来解析输入文本Wang et al.（2019）。

The advent of foundation models contributes to the development of such adaptive, on-the-fly navigation technologies by enhancing the understanding of human language instructions and the visual interpretation of environmental information. More detailed explanations of representative VLN research are provided in 6.2.2.
基础模型的出现通过增强对人类语言指令的理解和对环境信息的视觉解释，促进了这种自适应、飞行中导航技术的发展。代表性VLN研究的更详细解释见6.2.2。

An accumulating body of evidence suggests that recent VLMs and LLMs have promising capabilities for symbolic task planning (e.g., what-to-do). However, each task requires low-level control policy (e.g., how-to-do) to achieve successful interaction between the environment. While reinforcement learning and imitation learning are promising approach to learn policies in a data-driven manner, another promising approach is to obtain the strategy directly from humans through on-site demonstration, an approach called Learning-from-Observation Wake et al. (2021a); Ikeuchi et al. . In this section, we introduce a study where we employ ChatGPT for task planning and enrich the plan by parameterizing it with affordance information to facilitate effective and precise execution (Fig. 13).
越来越多的证据表明，最近的VLM和LLMs具有很好的符号任务规划能力（例如，要做什么）。然而，每个任务需要低级控制策略（例如，如何做），以实现环境之间的成功互动。虽然强化学习和模仿学习是以数据驱动的方式学习策略的有前途的方法，但另一种有前途的方法是通过现场演示直接从人类获得策略，这种方法称为从观察中学习Wake et al.（2021 a）;Ikeuchi etal.在本节中，我们介绍了一项研究，其中我们使用ChatGPT进行任务规划，并通过使用示能信息对其进行参数化来丰富计划，以促进有效和精确的执行（图13）。

The pipeline was composed of two modules: task planning and parameterization. In task planning, the system is fed with language instructions and the description of the working environment. These instructions, along with a predefined set of robot actions and output specifications, are compiled into a comprehensive prompt provided to ChatGPT, which then generates a sequence of decomposed tasks with their textual descriptions (Fig. 13; left pane). Notably, we employ a few-shot approach, meaning ChatGPT is not trained on this task, offering an advantage in applicability as it eliminates the need for hardware-dependent data collection and model training. Additionally, the textual descriptions in the output enable the user to check and adjust the results as necessary, which is a crucial feature for a safe and robust operation. Fig. 14 shows the qualitative results conducted for an agentic simulation on top of VirtualHome Puig et al. (2018). The results demonstrate a reasonable task plan and its flexibility in adjusting outputs, indicating the broad applicability of our approach.
管道由两个模块组成：任务规划和参数化。在任务规划中，系统被输入语言指令和工作环境的描述。这些指令，沿着一组预定义的机器人动作和输出规范，被编译成一个提供给ChatGPT的综合提示，然后生成一系列分解任务及其文本描述（图13;左窗格）。值得注意的是，我们采用了少量的方法，这意味着ChatGPT不会在此任务上进行训练，这在适用性方面具有优势，因为它消除了对依赖于硬件的数据收集和模型训练的需求。此外，输出中的文本描述使用户能够根据需要检查和调整结果，这是安全和稳健操作的关键功能。图 图14显示了在VirtualHomePuig et al.（2018）之上进行的代理模拟的定性结果。结果表明，一个合理的任务计划和调整输出的灵活性，表明我们的方法的广泛适用性。

While the task planner guarantees coherency between the task sequences, successful operation in reality requires detailed parameters. For example, grasp type is crucial for carrying a container while spilling out the content, such a parameter is often ignored in a simulators (see Fig. 14 in grasping a pie). In our robot system, therefore, users are asked to demonstrate each action visually (Fig. 13; right pane). The tasks had predefined parameters necessary for execution, which our vision system extracts from the videos Wake et al. (2021b). Notably, our robotic system is not designed for exact replication of human motions (i.e., teleoperation) but rather to handle varying real-world conditions, such as changes in object locations. Hence, the parameters extracted from human demonstrations encompass not precise motion paths but affordance information that dictates effective environmental movement (e.g., waypoints for collision avoidance Wake et al. (2023a), grasp types Wake et al. (2023e), and upper-limbs postures Sasabuchi et al. (2021); Wake et al. (2021a)). The posture of the upper limbs is critical in robots with high degrees of freedom and is designed to assume predictable postures for humans coexisting with the operational robot. The task sequence endowed with affordances is transformed into a sequence of reusable robot skills acquired through reinforcement learning and executed by the robot Takamatsu et al. (2022). 重试    错误原因

LLM-empowered task planning can be extended to a more versatile robotic system by integrating it with VLMs. Here, we show an example where we use the GPT-4V(ision) to broaden the aforementioned task planner in a multimodal input context (Fig. 15), a human performs actions that are intended to be replicated by the robot. In this paper, only part of the prompt is shown. The whole prompt is available at microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts.
LLM-授权的任务规划可以通过将其与VLM集成而扩展到更通用的机器人系统。在这里，我们展示了一个例子，其中我们使用GPT-4V（视觉）来扩展上述任务规划器在多模态输入环境中（图15），人类执行旨在由机器人复制的动作。本文仅显示部分提示。完整的提示可在microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts上找到。

This pipeline takes demonstration videos and text, then outputs a sequence of robot actions. A vision analyzer aims to understand the actions performed by humans in the video. We used GPT-4V and provided a prompt to generate text instructions in a style typical of human-to-human communication.Fig. 16 demonstrates how the usage of text input allows user to give feedback on GPT-4V’s recognition results for correction purposes. Such a feature, aiming at improving the accuracy of the recognition results, also enables more robust operation.
该管道获取演示视频和文本，然后输出一系列机器人动作。视觉分析器的目的是了解视频中人类所执行的动作。我们使用了GPT-4V，并提供了一个提示，以生成典型的人与人之间的交流风格的文本指令。图16演示了文本输入的使用如何允许用户对GPT-4V的识别结果给出反馈，以用于校正目的。这种旨在提高识别结果的准确性的特征还使得能够实现更鲁棒的操作。

Next, the scene analyzer compiles the expected work environment into the text information based on the instructions and the first frame of the video data (or an image of the environment). This environmental information includes a list of object names recognized by GPT-4V, the graspable properties of objects, and the spatial relationships between objects. Although these computational processes are a black box within GPT-4V, the information is output based on the knowledge of GPT-4V and the image/text input. Fig. 17 shows the example outputs of our scene analyzer. As shown in the figure, GPT-4V successfully selects the objects that are related to the manipulation. For example, a table is included in the output when the human is relocating a spam container on the table, while the table is ignored for the fridge opening task. These results suggest that the scene analyzer encodes the scene information with respect to the human’s actions. We prompted GPT-4V to explain the results of the object selection process and the reasons behind those choices. In practice, we found this approach resulted in reasonable outputs. Finally, based on the given text instructions and environmental information, the task planner outputs a sequence of tasks Wake et al. (2023c).
接下来，场景分析器基于指令和视频数据的第一帧（或环境的图像）将预期的工作环境编译成文本信息。该环境信息包括GPT-4V所识别的对象名称的列表、对象的可抓取属性以及对象之间的空间关系。虽然这些计算过程是GPT-4V中的黑盒，但信息是基于GPT-4V的知识和图像/文本输入输出的。图17显示了我们的场景分析器的示例输出。如图所示，GPT-4V成功地选择了与操作相关的对象。例如，当人类在桌子上重新定位垃圾邮件容器时，输出中包括桌子，而对于冰箱打开任务，忽略该桌子。这些结果表明，场景分析器编码的场景信息相对于人类的行动。 我们促使GPT-4V解释对象选择过程的结果以及这些选择背后的原因。在实践中，我们发现这种方法产生了合理的输出。最后，基于给定的文本指令和环境信息，任务规划器输出任务序列Wake等人（2023 c）。

Image-language understanding is a task that involves the interpretation of visual content in a given image with language and the generation of associated linguistic descriptions. This task is critical to the development of AI agents that can interact with the world in a more human-like manner. Some of most popular ones are image captioning Lin et al. (2014); Sharma et al. (2018); Young et al. (2014); Krishna et al. (2016), referring expression Yu et al. (2016); Karpathy et al. (2014), and visual question answering Antol et al. (2015); Ren et al. (2015); Singh et al. (2019).
图像语言理解是一项任务，涉及到在一个给定的图像与语言的视觉内容的解释和相关的语言描述的生成。这项任务对于开发能够以更人性化的方式与世界交互的人工智能代理至关重要。一些最流行的是图像字幕Lin et al.（2014）; Sharma et al.（2018）; Young et al.（2014）; Krishna et al.（2016），指涉表达Yu et al.（2016）; Karpathy et al.（2014）和视觉问答Antol et al.（2015）; Ren et al.（2015）; Singh et al.（2019）。

More recently, knowledge-intensive Visual Question Answering tasks such as OKVQA Marino et al. (2019), KB-VQA Wang et al. (2015), FVQA Wang et al. (2017), and WebQA Chang et al. (2021) have been introduced. Multimodal agents should capable of identifying objects in an image, comprehending their spatial relationships, generating accurate descriptive sentences about the scene, and utilizing reasoning skills to handle knowledge-intensive visual reasoning. This requires not just object recognition capabilities, but also a deep understanding of spatial relationships, visual semantics, and the ability to map these visual elements to linguistic constructs with integration of the world knowledge.
最近，引入了知识密集型的视觉问题分类任务，如OKVQAMarino et al.（2019），KB-VQAWang et al.（2015），FVQAWang et al.（2017）和WebQAChang et al.（2021）。多模态智能体应该能够识别图像中的对象，理解它们的空间关系，生成关于场景的准确描述性句子，并利用推理技能来处理知识密集型视觉推理。这不仅需要对象识别能力，还需要对空间关系、视觉语义的深刻理解，以及将这些视觉元素映射到语言结构并整合世界知识的能力。

• •

  Knowledge-Intensive Models: As introduced in INK Park et al. (2022), and KAT Gui et al. (2022a), an intensive neural knowledge task that incorporates required knowledge annotated by humans to support knowledge-intensive retrieval task.
  知识密集型模型：如INKPark等人（2022）和KATGui等人（2022 a）所介绍的，一种密集的神经知识任务，它包含了人类注释的所需知识，以支持知识密集型检索任务。

• •

  Multimodal-Agents: There has been a growing interest in multimodal language models like Chameleon Lu et al. (2023) and MM-React Yang et al. (2023c).
  多模式代理： 人们对多模态语言模型越来越感兴趣， ChameleonLu等人（2023）和MM-ReactYang等人（2023 c）。

• •

  Visual Instruction Tuning: VCLGui et al. (2022b), Mini-GPT4 Zhu et al. (2023), MPLUG-OWL Ye et al. (2023b), LSKD Park et al. (2023c) generate image-level instruction tuning dataset.
  视觉指令调整：VCLGui等人（2022 b）、Mini-GPT 4Zhu等人（2023）、MPLUG-OWLYe等人（2023 b）、LSKDPark等人（2023 c）生成图像级指令调整数据集。

Recognizing task directives and taking action has been a fundamental challenge in interactive AI and natural language processing for decades. With the recent advances in deep learning, there is a growing interest in studying these areas jointly to improve human-agent collaboration. We identify three specific directions, among others, to improve language-grounded agents:
几十年来，识别任务指令并采取行动一直是交互式人工智能和自然语言处理的基本挑战。随着深度学习技术的发展，人们越来越有兴趣联合研究这些领域，以改善人-智能体协作。我们确定了三个具体的方向，以改善语言为基础的代理：

• •

  Tool use and querying from knowledge bases. This direction emphasizes the importance of integrating external knowledge bases, web search, or other helpful tools into the reasoning processes of AI agents. By leveraging structured and unstructured data from various sources, agents can enhance their understanding and provide more accurate and context-aware responses. Furthermore, it fosters the agent’s ability to proactively seek out information when faced with unfamiliar scenarios or queries, ensuring more comprehensive and informed responses. Examples include Toolformer Schick et al. (2023) and Retrieve What You Need Wang et al. (2023g).
  工具使用和知识库查询。该方向强调了将外部知识库、网络搜索或其他有用工具集成到人工智能代理的推理过程中的重要性。通过利用来自各种来源的结构化和非结构化数据，代理可以增强其理解并提供更准确和上下文感知的响应。此外，它还能培养代理在遇到不熟悉的情况或查询时主动寻找信息的能力，从而确保做出更全面、更明智的响应。例子包括ToolformerSchick等人（2023）和Retrieve What You NeedWang等人（2023 g）。

• •

  Improved agent reasoning and planning. Enhancing the agent’s ability to reason and plan is pivotal for effective human-agent collaboration. This involves the development of models that can understand complex instructions, infer user intentions, and predict potential future scenarios. This can be accomplished by asking the agent to reflect on past actions and failures as in ReAct Yao et al. (2023a), or by structuring the agent thought process as a form of search Yao et al. (2023b). By simulating different outcomes and assessing the ramifications of various actions, agents can make more informed context-aware decisions.
  改进了座席的推理和规划。提高智能体的推理和规划能力是有效的人-智能体协作的关键。这涉及到开发能够理解复杂指令、推断用户意图和预测潜在未来场景的模型。这可以通过要求代理反思过去的行动和失败来实现，如ReActYao et al.（2023 a），或者通过将代理思维过程结构化为搜索的形式Yao et al.（2023 b）。通过模拟不同的结果并评估各种行动的后果，代理可以做出更明智的上下文感知决策。

• •

  Incorporating system and human feedback. AI agents can frequently operate in two primary contexts: environments that provide explicit signals about the effectiveness of their actions (system feedback), and settings where they collaborate with humans who can offer verbal critiques (human feedback). This direction underscores the need for adaptive learning mechanisms that allow agents to refine their strategies and rectify mistakes, such as in AutoGen Wu et al. (2023). The ability to continuously learn and adapt from diverse feedback sources ensures that agents remain helpful and aligned for user needs.
  系统和人的反馈。人工智能代理通常可以在两种主要环境中工作：一种是提供关于其行为有效性的明确信号的环境（系统反馈），另一种是与能够提供口头批评的人类合作的环境（人类反馈）。这一方向强调了自适应学习机制的必要性，这些机制允许智能体改进其策略并纠正错误，例如AutoGenWu et al.（2023）。不断学习和适应不同反馈源的能力确保了代理保持对用户需求的帮助和一致性。

Recognizing and understanding agent content and natural language has been a fundamental challenge in interactive AI and natural language processing for decades. With the recent advances in deep learning, there is a growing interest in studying these two areas jointly for deep understanding of both agent planning or human feedback for knowledge-inference and natural language generation. These are the key components of many human-machine-interaction agents, such as “AutoGen"Wu et al. (2023) and “Retrieve What You Need"Wang et al. (2023g).
几十年来，识别和理解智能体内容和自然语言一直是交互式人工智能和自然语言处理的基本挑战。随着深度学习的最新进展，人们越来越有兴趣联合研究这两个领域，以深入理解智能体规划或人类反馈，从而进行知识推理和自然语言生成。这些是许多人机交互代理的关键组件，例如“AutoGen”Wu et al.（2023）和“满足您的需求”Wang et al.（2023 g）。

Furthermore, the creation of LLM Agents that can be trained to effectively follow human instructions has become an important area of research. Initial models used human feedback to train a proxy reward model to simulate human preferences, through a process known as Reinforcement Learning with Human Feedback (RLHF) Ouyang et al. (2022). This process produced models such as InstructGPT and ChatGPT. In order to more efficiently train instruction-following LLM agents without needing human labels, researchers developed a more efficient method for instruction-tuning that trains the LLM agent directly on instruction/response pairs, either generated by humans like Dolly 2.0 ⁵⁵5Dolly 2.0 blogpost link or automatically from LLMs like Alpaca Taori et al. (2023). We show the overall Alpaca training pipeline in Figure 28.
此外，创建LLM智能体，可以训练它们有效地遵循人类的指令，已经成为一个重要的研究领域。最初的模型使用人类反馈来训练代理奖励模型，以模拟人类的偏好，通过一个被称为人类反馈强化学习（RLHF）的过程，欧阳等人（2022）。这一过程产生了诸如InstructGPT和ChatGPT的模型。为了在不需要人类标签的情况下更有效地训练遵循LLM的智能体，研究人员开发了一种更有效的方法来训练LLM智能体，该方法直接在指令/响应对上训练LLM智能体，这些指令/响应对可以由人类（如Dolly 2.0 ⁵⁵5Dolly 2.0 blogpost link ）生成，也可以由LLMs自动生成，如AlpacaTaori等人（2023）。我们在图28中展示了整个羊驼训练流程。

Despite the growing adoption of conversational and self-feedback systems, these forms of AI still do not perform well with regard to generating factually correct responses from their own implicit knowledge and therefore often use external tools like web search and knowledge retrieval mechanisms at inference-time to augment their response as a consequence. Addressing this would help create more engaging experiences for users in many real-life applications. In social conversations (such as those on social media platforms like Instagram and Facebook), or with Q+A websites (such as Ask or Quora), people usually engage with others through a series of comments and by web-searching for information and knowledge relevant to the discussion. Thus, the task of generating conversational turns in this context is not to simply bootstrap upon traditional NLP models and tasks, but to use agents to generate dialogue through intelligent behaviors that reflect knowledge search and acquisition Peng et al. (2023). In this way, intelligent agents for NLP tasks extends the task description and improves upon the interpretability of the response by adding an explicit knowledge search and retrieval step during dialogue. Incorporating these web search and retrieval agents as feedback during dialogue will help to engage further and deeper the social interactions between humans and agents Wang et al. (2023e). As the Fig 29 showed, we introduced a new modeling paradigm for transformer language models that detects and extracts important logical structures and information from input texts and then integrates them into the input embeddings through carefully designed multi-layer hierarchical logical projections to infuse logical structures into pre-trained language models as one kind of NLP agent. Wang et al. (2023e) propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade all existing transformer language models into logical transformers to consistently boost their performance. The proposed logical transformer agent consistently achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts. To human users, it is often these aspects that are more important for delivering a meaningful and interesting conversation via a agent-based coordination between dialogue and information retrieval. Delving deep into natural language processing, this topic will discuss the advancements and leader-board in making LLMs more agentic and better suited for various language-centered tasks.
尽管越来越多地采用对话和自我反馈系统，但这些形式的人工智能在从自己的隐含知识中生成事实上正确的响应方面仍然表现不佳，因此经常在推理时使用外部工具，如网络搜索和知识检索机制来增强其响应。解决这一问题将有助于在许多现实生活中的应用程序中为用户创造更具吸引力的体验。在社交对话中（例如在Instagram和Facebook等社交媒体平台上的对话），或者在问答网站上（例如Ask或Quora），人们通常通过一系列评论和网络搜索与讨论相关的信息和知识来与他人互动。 因此，在这种情况下生成会话轮的任务不是简单地在传统的NLP模型和任务上引导，而是使用代理通过反映知识搜索和获取的智能行为来生成对话。 通过这种方式，用于NLP任务的智能代理扩展了任务描述，并通过在对话期间添加明确的知识搜索和检索步骤来提高响应的可解释性。将这些网络搜索和检索代理作为对话期间的反馈，将有助于进一步和更深入地参与人类和代理之间的社会互动。  如图29所示，我们为Transformer语言模型引入了一种新的建模范式，该范式从输入文本中检测和提取重要的逻辑结构和信息，然后通过精心设计的多层分层逻辑投影将它们集成到输入嵌入中，以将逻辑结构注入到预先训练的语言模型中，作为一种NLP代理。Wang等人（2023 e）提出了一种新的方法，通过逻辑检测、逻辑映射和层次逻辑投影的组合，为Transformer语言模型构建逻辑感知的输入嵌入，然后开发相应的新建模范式，可以将所有现有的Transformer语言模型升级为逻辑transformer，以持续提高其性能。 所提出的逻辑Transformer代理一致地实现上级的性能超过其基线Transformer模型，通过更深入地理解文本的逻辑结构。对于人类用户来说，这些方面通常对于通过对话和信息检索之间基于代理的协调来提供有意义和有趣的对话更重要。深入研究自然语言处理，本主题将讨论在使LLMs更具代理性和更适合各种以语言为中心的任务方面的进步和领先地位。

An open-domain question answering (QA) system usually follows a retrieve-then-read paradigm, in which a retriever is used to retrieve relevant passages from a large corpus, and then a reader generates answers based on the retrieved passages and the original question. In  Wang et al. (2023g), we propose a simple and novel mutual learning framework to improve the performance of retrieve-then-read-style models via an intermediate module named the knowledge selector agent, which we train with reinforcement learning. The fine-grained knowledge selector into the retrieve-then-reader paradigm, whose goal is to construct a small subset of passages which retain question-relevant information. As showed in Figure 30, The knowledge selector agent is trained as a component of our novel mutual learning framework, which iteratively trains the knowledge selector and the reader. We adopt a simple and novel approach employing policy gradients to optimize the knowledge selector agnet, using feedback from the reader to train it to select a small and informative set of passages. This approach avoids brute-force search or manually-designed heuristics, without requiring any annotated query-document pairs for supervision. We show that iteratively training the reader and the knowledge selector agent leads to better predictive performance on some public open-domain question answering benchmarks.
开放域问答系统通常遵循检索-阅读模式，即检索器从大型语料库中检索相关段落，然后读者根据检索到的段落和原始问题生成答案。在Wang et al.（2023 g）中，我们提出了一个简单而新颖的相互学习框架，通过一个名为知识选择器代理的中间模块来提高检索然后阅读风格模型的性能，我们用强化学习训练它。细粒度的知识选择器进入检索然后读者范式，其目标是构建一个小的子集的段落保留问题相关的信息。 如图30所示，知识选择器代理被训练为我们的新的相互学习框架的一个组件，该框架迭代地训练知识选择器和阅读器。我们采用了一种简单而新颖的方法，采用政策梯度来优化知识选择器agnet，使用读者的反馈来训练它选择一个小的和翔实的段落集。这种方法避免了强制搜索或手动设计的启发式方法，不需要任何带注释的查询-文档对进行监督。我们表明，迭代训练读者和知识选择器代理导致更好的预测性能在一些公共开放领域的问答基准。

For our competition, we will release a benchmark, the CuisineWorld benchmark, which includes a text interface that includes extendable task definition files, and an interface for multi-agent interaction, and human-machine interactions. We introduce the gaming interaction task in which the goal is to generate relevant, appropriate, multi-agent collaboration strategies that can maximize collaboration efficiency. We evaluate the collaboration efficiency with the proposed evaluation metric: CoS.
对于我们的比赛，我们将发布一个基准，CuisineWorld基准，其中包括一个包含可扩展任务定义文件的文本界面，以及一个用于多代理交互和人机交互的界面。我们引入了博弈交互任务，其目标是生成相关的、合适的、多Agent协作策略，以最大化协作效率。我们使用所提出的评估指标CoS来评估协作效率。

The “CuisineWorld" dataset was collected by Microsoft, UCLA, and Stanford University. The goal of the competition is to explore how different, existing and novel, grounded-LLM and interactive techniques perform with this benchmark and establish strong baselines for the task of multi-agent gaming infrastructure.
“CuisineWorld”数据集由微软、加州大学洛杉矶分校和斯坦福大学收集。比赛的目标是探索不同的，现有的和新颖的，接地的LLM和交互式技术如何与此基准一起执行，并为多智能体游戏基础设施的任务建立强大的基线。

The dataset of CuisineWorld includes:
CuisineWorld的数据集包括：

- A selection of well-defined multi-agent collaboration tasks.
- 一系列定义良好的多智能体协作任务。
- An API system to facilitate agent interactions.
- 一个API系统，用于促进代理交互。
- An automatic evaluation system.
- 一个自动评估系统。

(The link for downloading the dataset will soon be made available and this article will be updated to include it here.)
(The用于下载数据集的链接将很快提供，本文将更新为在此处包括该链接。）

• •

  We provide a dataset and related the benchmark, called Microsoft MindAgent and and correspondingly release a dataset “CuisineWorld” to the to the research community.
  我们提供了一个数据集和相关的基准，称为Microsoft MindAgent，并相应地发布了一个数据集“CuisineWorld”给研究社区。

• •

  We will provide benchmarks to evaluate and rank the submitted “MindAgent" algorithms. We will also provide baseline results generated using popular infrastructures.
  我们将提供基准来评估和排名提交的“MindAgent”算法。我们还将提供使用常用基础架构生成的基准结果。

The quality of multi-agent collaboration efficiency is determined by the new “cos" auto-metric (from MindAgent Gong et al. (2023a)). The final rating of out metric is calculated as an average over the evaluated collaboration efficiency metrics of the multi-agent system on all tasks. Human evaluators will be asked to rate individual responses as well as provide subjective judgement of the engagement, breadth and an overall quality of the users’ interactions with the agents.
多智能体协作效率的质量由新的“cos”自动度量确定（来自MindAgentGong等人（2023 a））。输出度量的最终评级被计算为多代理系统在所有任务上的所评估的协作效率度量的平均值。将要求人类评估人员对个人的反应进行评级，并对用户与代理的互动的参与度、广度和整体质量提供主观判断。

• •

  Automated Evaluation. We plan to release a leaderboard, starting on the release date (TBA), registered participants will be asked to submit their results on the task associated with the dataset “CuisineWorld" (our publicly released dataset for the leaderboard). Submission of results will be closed on the end date (TBA). Each team will be required to submit their generated results on the testing set for automated evaluation of the “cos" metric.
  自动评估。我们计划发布排行榜，从发布日期（TBA）开始，注册参与者将被要求提交与数据集“CuisineWorld”（我们公开发布的排行榜数据集）相关的任务结果。结果提交将在结束日期（TBA）截止。每个团队将被要求提交他们在测试集上生成的结果，以自动评估“cos”指标。

• •

  Human Evaluation on our leaderboard. The leaderboard participants will need to provide a submission file generated by evaluation scripts locally. We will use the evalAI system to check the submission file and optionally rerun the code for top challenge contenders. Therefore, teams must also submit their code with a Readme file on how to run their code. Human evaluation will be performed by the organization team.
  在我们的排行榜上的人类评价。排行榜参与者需要提供一个由本地评估脚本生成的提交文件。我们将使用evalAI系统来检查提交文件，并选择性地检查顶级挑战者的代码。因此，团队还必须提交他们的代码，并附上一个关于如何运行代码的自述文件。将由组织团队进行人因评价。

• •

  Winner Announcement. We will make an announcement of the winners and post the final ratings of the submissions on our leaderboard.
  赢家宣布。我们将公布获奖者，并在我们的排行榜上公布提交的最终评级。