Do Embodied Agents Dream of Pixelated Sheep?:
Embodied Decision Making using Language Guided World Modelling
嵌入式代理是否梦见像素化的羊？：使用语言引导的世界建模进行具体化决策

Textual knowledge can be used to improve generalization in reinforcement learning through environment descriptions (Branavan et al., 2011; Zhong et al., 2020; Hanjie et al., 2021) or language instructions (Chevalier-Boisvert et al., 2019; Anderson et al., 2018; Ku et al., 2020; Shridhar et al., 2020). However, task specific textual knowledge is expensive to obtain, prompting the use of web queries (Nottingham et al., 2022) or models pretrained on general world knowledge (Dambekodi et al., 2020; Suglia et al., 2021; Ichter et al., 2022; Huang et al., 2022b; Song et al., 2022b).
文本知识可以通过环境描述（Branavan 等，2011 年；Zhong 等，2020 年；Hanjie 等，2021 年）或语言指令（Chevalier-Boisvert 等，2019 年；Anderson 等，2018 年；Ku 等，2020 年；Shridhar 等，2020 年）来改善强化学习中的泛化能力。然而，特定任务的文本知识获取成本高昂，促使人们使用网络查询（Nottingham 等，2022 年）或预先训练的通用世界知识模型（Dambekodi 等，2020 年；Suglia 等，2021 年；Ichter 等，2022 年；Huang 等，2022 年 b；Song 等，2022 年 b）。

LLMs can also be used as an external knowledge source by prompting or finetuning them to generate action plans. However, by default, the generated plans are not grounded in environment dynamics and constraining output can harm model performance, both of which lead to subpar performance of out-of-the-box LLMs on decision-making tasks (Valmeekam et al., 2022). Existing work that uses LLMs for generating action plans focuses on methods for grounding language in environment states (Ichter et al., 2022; Huang et al., 2022b; Song et al., 2022b), or improving LLM plans through more structured output (Singh et al., 2022; Liang et al., 2022b). In this work, we focus on using LLMs for exploration rather than directly generating action plans.
LLMs也可以被用作外部知识源，通过提示或微调它们来生成行动计划。然而，默认情况下生成的计划并没有基于环境动态，并且限制输出可能会损害模型性能，这两者都会导致LLMs在决策任务中表现不佳（Valmeekam等人，2022年）。现有的使用LLMs生成行动计划的工作侧重于将语言与环境状态相结合的方法（Ichter等人，2022年；Huang等人，2022b年；Song等人，2022b年），或通过更结构化的输出改进LLM计划（Singh等人，2022年；Liang等人，2022b年）。在本研究中，我们重点关注使用LLMs进行探索，而不是直接生成行动计划。

Tam et al. (2022) and Mu et al. (2022) recently demonstrated that language is a meaningful state abstraction when used for exploration. Additionally, Tam et al. (2022) experiment with using LLM latent representations of state descriptions for novelty exploration, relying on pretrained LLM encodings to detect novel textual states. To the best of our knowledge, we are the first to apply language-assisted decision-making to exploration by using LLMs to predict and verify environment dynamics through experience.
Tam 等人（2022 年）和 Mu 等人（2022 年）最近表明，当用于探索时，语言是一种有意义的状态抽象。此外，Tam 等人（2022 年）尝试使用状态描述的潜在表示进行新颖性探索，依赖于预训练的编码来检测新颖的文本状态。据我们所知，我们是首个通过使用语言辅助决策来进行探索的人，通过经验来预测和验证环境动态。

Without grounding, LLMs often fail to reason about real world dynamics (Bisk et al., 2020). Instruction following tasks have been a popular testbed for language grounding (Chevalier-Boisvert et al., 2019; Anderson et al., 2018; Ku et al., 2020; Shridhar et al., 2020) prompting many improvements to decision making conditioned on language instructions (Yu et al., 2018; Lynch & Sermanet, 2020; Nottingham et al., 2021; Suglia et al., 2021; Kuo et al., 2021; Zellers et al., 2021; Song et al., 2022a; Blukis et al., 2022). Other prior work used environment interactions to ground responses from question answering models in environment state (Gordon et al., 2018; Das et al., 2018) or physics (Liu et al., 2022). Finally, Ammanabrolu & Riedl (2021) learn a grounded textual world model from environment interactions to assist an RL agent in planning and action selection. In this work, our DECKARD agent also uses a type of textual world model but it is obtained few-shot from an LLM and then grounded in environment dynamics by verifying hypotheses through interaction.
缺乏基础支撑，LLMs往往无法推理真实世界的动态（Bisk等人，2020年）。遵循指令的任务已成为语言基础的流行测试基地（Chevalier-Boisvert等人，2019年；Anderson等人，2018年；Ku等人，2020年；Shridhar等人，2020年），促使许多改进以便根据语言指令进行决策（Yu等人，2018年；Lynch和Sermanet，2020年；Nottingham等人，2021年；Suglia等人，2021年；Kuo等人，2021年；Zellers等人，2021年；Song等人，2022a年；Blukis等人，2022年）。其他先前的工作使用环境交互来使问题回答模型的响应根植于环境状态（Gordon等人，2018年；Das等人，2018年）或物理学（Liu等人，2022年）。最后，Ammanabrolu＆Riedl（2021年）从环境交互中学习了一个基于文本的世界模型，以协助RL代理进行规划和动作选择。在这项工作中，我们的DECKARD代理也使用一种文本世界模型，但它是从一个LLM中获得的，并通过验证假设进行交互来使其根植于环境动态。

Modular RL proposes to learn several independent policies in a composable way to facilitate training and generalization (Simpkins & Isbell, 2019). Ammanabrolu et al. (2020) and Patil et al. (2020) demonstrate how modular policies can improve exploration by reducing policy horizons, the former using the text-based game Zork and the latter using Minecraft. We implement modularity for Minecraft by finetuning a pretrained transformer policy with adapters, a technique recently implemented for RL by Liang et al. (2022a) for multi-task robotic policies.
模块化强化学习提出以可组合的方式学习多个独立策略，以促进训练和泛化（Simpkins＆Isbell，2019）。Ammanabrolu 等人（2020 年）和 Patil 等人（2020 年）展示了模块化策略如何通过减少策略水平来改善探索，前者使用基于文本的游戏 Zork，后者使用 Minecraft。我们通过使用适配器对预训练的转换器策略进行微调来实现 Minecraft 的模块化，这是最近由 Liang 等人（2022a 年）为多任务机器人策略实现的一种技术。

Minecraft is a vast open-ended world with complex dynamics and sparse rewards. Crafting items in the Minecraft technology tree has long been considered a challenging task for reinforcement learning, requiring agents to overcome extremely delayed rewards and difficult exploration (Skrynnik et al., 2021; Patil et al., 2020; Hafner et al., 2023). This is partially due to the scarcity of items in the environment, but also due to the depth of some items in the game’s technology tree. The purpose of our work is to overcome the latter of these two difficulties by better learning and navigating Minecraft’s technology tree.
《我的世界》是一个庞大的开放式世界，具有复杂的动态和稀缺的奖励。长期以来，制作《我的世界》科技树中的物品一直被认为是强化学习的一项具有挑战性的任务，需要代理人克服极端延迟的奖励和艰难的探索。这部分是由于环境中物品的稀缺性，但也是由于游戏科技树中一些物品的深度。我们工作的目的是通过更好地学习和导航《我的世界》的技术树来克服这两个困难之一。

Several existing agents overcome the problem of item scarcity in Minecraft by simplifying environment parameters such as action duration (Patil et al., 2020) or block break time (Hafner et al., 2023), making comparison between methods difficult. For this reason we compare minimally to other Minecraft agents (see Table 2), focusing our evaluation on the benefits of LLM-guided exploration with DECKARD. We use the video pretrained (VPT) Minecraft agent (Baker et al., 2022) as a starting point for exploration and finetuning, and we use the Minedojo implementation of the Minecraft Environment (Fan et al., 2022).
几种现有的代理通过简化环境参数（例如动作持续时间（Patil等人，2020）或方块破坏时间（Hafner等人，2023））来克服《我的世界》中物品稀缺的问题，使得方法之间的比较变得困难。出于这个原因，我们仅与其他《我的世界》代理进行了最小程度的比较（见表2），并将评估重点放在 DECKARD 引导探索的优点上。我们使用视频预训练（VPT）《我的世界》代理（Baker等人，2022）作为探索和微调的起点，并使用 Minedojo 实现的《我的世界》环境（Fan等人，2022）。

Rather than train a single policy with sparse rewards, modular RL advocates learning compositional policy modules (Simpkins & Isbell, 2019). DECKARD automatically discovers subgoals in Minecraft—each of which maps to an independently trained policy module—and learns a DAG of dependencies (the AWM) to transition between subgoals. Policy modules are trained in an environment modeled by a POMDP with states $s\in\mathcal{S}$, obseravtions $o\in O$, actions $a\in\mathcal{A}$, and environment dynamics $\mathcal{T}:\mathcal{S},\mathcal{A}\to\mathcal{S}^{\prime}$. These elements are common between modules, but each subgoal defines different initial states $\mathcal{S}_{0}$ and observations $O_{0}$, terminal states $\mathcal{S}_{t}$, and reward functions $\mathcal{R}:\mathcal{S},\mathcal{A}\to\mathbb{R}$, according to the particular subgoal. $\mathcal{S}_{0}$ and $O_{0}$ are defined by the current subgoal’s parents in the DAG, and $\mathcal{S}_{t}$ and $\mathcal{R}$ are defined by the current subgoal. For example, the craft wooden pickaxe subgoal has parents craft planks and craft stick, so $\mathcal{S}_{0}$ includes these items in the agent’s starting inventory. This subgoal recieves a reward and terminates when a wooden pickaxe is added to the agent’s inventory. Section 5 and Appendix B provide more details.
与训练稀疏奖励的单一策略不同，模块化RL倡导学习组合策略模块（Simpkins＆Isbell，2019）。DECKARD在Minecraft中自动发现子目标——每个子目标都映射到一个独立训练的策略模块——并学习依赖关系的DAG（AWM）以在子目标之间进行转换。策略模块在一个由POMDP建模的环境中进行训练，该环境具有状态 $s\in\mathcal{S}$ ，观察 $o\in O$ ，动作 $a\in\mathcal{A}$ 和环境动态 $\mathcal{T}:\mathcal{S},\mathcal{A}\to\mathcal{S}^{\prime}$ 。这些元素在模块之间是共同的，但每个子目标根据特定子目标定义不同的初始状态 $\mathcal{S}_{0}$ 和观察 $O_{0}$ ，终止状态 $\mathcal{S}_{t}$ 和奖励函数 $\mathcal{R}:\mathcal{S},\mathcal{A}\to\mathbb{R}$ 。 $\mathcal{S}_{0}$ 和 $O_{0}$ 由DAG中当前子目标的父级定义， $\mathcal{S}_{t}$ 和 $\mathcal{R}$ 由当前子目标定义。例如，制作木镐的子目标具有制作木板和制作棍子的父级，因此 $\mathcal{S}_{0}$ 在代理的起始库存中包括这些物品。当木镐添加到代理的库存中时，该子目标接收奖励并终止。第5节和附录B提供了更多细节。

Due to the compositionality of modular RL, individual modules can be chained together to achieve complex tasks. In our case, given a goal state $s_{g}$, we use the subgoal DAG to create a path from our current state to $s_{g}$, $[s_{0},s_{1},...,s_{g}]$, where each $s$ represents the terminal state for a subgoal. By chaining together sequences of subgoal modules, we can successfully navigate to connected portions of the currently discovered DAG and reach arbitrary goal states.
由于模块化 RL 的组合性，可以将单独的模块链在一起以实现复杂的任务。在我们的情况下，给定一个目标状态 $s_{g}$ ，我们使用子目标 DAG 来创建从当前状态到 $s_{g}$ ， $[s_{0},s_{1},...,s_{g}]$ 的路径，其中每个 $s$ 表示子目标的终端状态。通过链式连接子目标模块的序列，我们可以成功导航到当前发现的 DAG 的连接部分并达到任意目标状态。

Large language models (LLM) are trained with a language modeling objective to maximize the likelihood of training data from large text corpora. As LLMs have grown in size and representational power, they have seen success on various downstream tasks by simply modifying their input, referred to as prompt engineering (Brown et al., 2020). Recent applications of LLMs to decision-making have relied partially or entirely on prompt engineering for their action planning (Ichter et al., 2022; Song et al., 2022b; Huang et al., 2022b; Singh et al., 2022; Liang et al., 2022b). We follow this pattern to extract knowledge from LLMs and construct our AWM. We prompt OpenAI’s Codex model (OpenAI, 2022) to generate DECKARD’s hypothesized AWM. Codex is trained to generate code samples from natural language. As with previous work (Singh et al., 2022; Liang et al., 2022b), we find that structured code output works well for extracting knowledge from LLMs. We structure LLM output by prompting Codex to generate a python dictionary of Minecraft item dependencies, which we then map to a graph of items and their dependencies (see Section 5.1 and Appendix A).
大型语言模型（LLM）是通过语言建模目标进行训练的，以最大化来自大型文本语料库的训练数据的可能性。随着LLMs在规模和表征能力上的增长，它们通过简单修改输入，在各种下游任务上取得了成功，这被称为提示工程（Brown等人，2020）。最近LLMs在决策方面的应用部分或完全依赖于提示工程来进行其行动规划（Ichter等人，2022年；Song等人，2022b年；Huang等人，2022b年；Singh等人，2022年；Liang等人，2022b年）。我们遵循这种模式，从LLMs中提取知识并构建我们的AWM。我们提示OpenAI的Codex模型（OpenAI，2022年）生成DECKARD假设的AWM。Codex被训练以从自然语言生成代码示例。与以前的工作一样（Singh等人，2022年；Liang等人，2022b年），我们发现结构化的代码输出对于从LLMs中提取知识很有效。我们通过提示Codex生成Minecraft物品依赖的Python字典来构造LLM输出，然后将其映射到物品及其依赖关系的图形（参见第5.1节和附录A）。

Our method, DECision-making for Knowledgable Autonomous Reinforcement-learning Dreamers (DECKARD), builds an Abstract World Model (AWM) of subgoal dependencies from state abstractions. We begin by assuming a textual state representation function $\phi:O\to X$. Textual state representations $x\in X$ make up the nodes for our AWM $G:X,E$ with directed edges $E$ defining the dependencies between $X$. We further constrain $G$ to a directed acyclic graph (DAG) so that the nodes of the DAG represent subgoals useful in navigating towards a target goal. In our experiments, we use the agent’s current inventory as $X$, a common component of the Minecraft observation space (Fan et al., 2022; Hafner et al., 2023).
我们的方法，DECision-making for Knowledgable Autonomous Reinforcement-learning Dreamers（DECKARD），从状态抽象构建子目标依赖的抽象世界模型（AWM）。我们首先假设一个文本状态表示函数 $\phi:O\to X$ 。文本状态表示 $x\in X$ 构成我们的 AWM $G:X,E$ 的节点，有指向的边 $E$ 定义了之间的依赖关系 $X$ 。我们进一步将 $G$ 限制为有向无环图（DAG），使得 DAG 的节点表示对导航向目标目标有用的子目标。在我们的实验中，我们使用代理的当前库存作为 $X$ ，这是 Minecraft 观察空间的一个常见组件（Fan 等人，2022年；Hafner 等人，2023年）。

We update $G$ from agent experience through environment exploration. When the agent experiences node $x_{t}$ for the first time, $G$ is updated by adding edges between the previous node $x_{t-1}$ and the new node $x_{t}$. When trying to reach a previously experienced node, DECKARD recovers the path from current node $x_{0}$ to the target node $x_{t}$ from the AWM. DECKARD then executes policies for each recovered node until it reaches the target goal.
通过环境探索，我们从代理经验中更新 $G$ 。当代理第一次经历节点 $x_{t}$ 时，通过在前一个节点 $x_{t-1}$ 和新节点 $x_{t}$ 之间添加边来更新 $G$ 。当尝试到达先前经历过的节点时，DECKARD 从当前节点 $x_{0}$ 到目标节点 $x_{t}$ 的路径从 AWM 中恢复。然后，DECKARD 执行每个恢复节点的策略，直到达到目标目标。

The setup so far (referred to in our experiments as “DECKARD (No LLM)”) allows the construction of a modular RL policy for navigating subgoals. However, the agent is still learning the AWM tabula-rasa. The key insight of DECKARD is that we can hypothesize the AWM with knowledge from an LLM. We use in-context learning, as described in Section 5.1, to predict $G$ from an LLM with predicted edges, $\hat{E}$. While acting in the environment, we verify or correct edges of $G$ and track the set of nodes that have been verified $V$ thus grounding the AWM hypothesized by the LLM in environment dynamics.
到目前为止的设置（在我们的实验中称为“DECKARD（无LLM）”）允许构建用于导航子目标的模块化 RL 策略。然而，代理仍在从头学习 AWM。 DECKARD 的关键见解是，我们可以使用LLM的知识假设 AWM。我们使用上下文学习，如 5.1 节所述，从具有预测边缘 $\hat{E}$ 的LLM中预测 $G$ 。在环境中行动时，我们验证或纠正 $G$ 的边缘，并跟踪已验证的节点集 $V$ ，从而将LLM对环境动态的假设 AWM 接地。

In our experiments, we predict the AWM using OpenAI’s Codex model (OpenAI, 2022) by prompting the LLM to generate recipes for Minecraft items. We prompt Codex to “Create a nested python dictionary containing crafting recipes and requirements for minecraft items” along with additional instructions about the dictionary contents and two examples: diamond pickaxe and diamond (see Appendix A). We iterate over 391 Minecraft items, generating recipes as well as tool requirements (mining stone requires a pickaxe) and workbench requirements (crafting a pickaxe requires a crafting table). The hypothesized AWM is generated at the start of training, so no forward passes of the LLM are necessary during training or inference. Table 1 shows the accuracy of the hypothesized un-grounded AWM.
在我们的实验中，我们使用OpenAI的Codex模型（OpenAI，2022年）来预测AWM，通过提示LLM为Minecraft物品生成配方。我们提示Codex“创建一个包含Minecraft物品的制作配方和要求的嵌套Python字典”，并附加有关字典内容的额外说明以及两个示例：钻石镐和钻石（见附录A）。我们迭代了391个Minecraft物品，生成配方以及工具要求（挖掘石头需要镐）和工作台要求（制作镐需要工作台）。假设的AWM在训练开始时生成，因此在训练或推断过程中不需要LLM的前向传播。表1显示了假设的未接地AWM的准确性。

Rather than train each module from scratch, we finetune transformer adapters for each module with an RL objective following the adapter architecture from Houlsby et al. (2019). We use the Video-Pretrained (VPT) Minecraft model as our starting policy (Baker et al., 2022). We chose to finetune VPT as it proved to be more sample efficient and more stable than training policies from scratch. Moreover, since VPT is pretrained on a variety of Minecraft skills, the non-finetuned VPT model explores the environment more thoroughly than a random agent. Our implementation of VPT finetuned with adapters is referred to as VPT-a.
不是每个模块都从头开始训练，而是我们使用强化学习目标对每个模块的 Transformer 适配器进行微调，遵循 Houlsby 等人的适配器架构（2019 年）。我们使用视频预训练（VPT）Minecraft 模型作为我们的起始策略（Baker 等人，2022 年）。我们选择微调 VPT，因为它被证明比从头开始训练的策略更具样本效率和稳定性。此外，由于 VPT 在各种 Minecraft 技能上预训练过，非微调的 VPT 模型比随机智能体更彻底地探索环境。我们使用适配器微调的 VPT 实现称为 VPT-a。

Adapters are especially well suited for modular finetuning due to their lightweight architecture (Liang et al., 2022a). In our agent, each subgoal module corresponds to one set of adapters and only contains 9.5 million trainable parameters, approximately 2% of the 0.5 billion parameter VPT model. This allows us to train a separate set of adapters for each subgoal and still keep all parameters in memory concurrently, a practical benefit of using adapters for modular, compositional RL policies.
由于其轻量级架构，适配器特别适用于模块化微调（Liang 等人，2022 年 a）。在我们的智能体中，每个子目标模块对应一个适配器集，并且仅包含 950 万可训练参数，约占 0.5 亿参数 VPT 模型的 2%。这使我们能够为每个子目标训练一个单独的适配器集，并仍然同时保留所有参数在内存中，这是使用适配器进行模块化、组合式强化学习策略的实际优势。

We use Minedojo’s Minecraft implementation for our experiments (Fan et al., 2022). As with VPT (Baker et al., 2022), our subgoal policies use a pixel only observation space and a large multi-discrete action space, while our overall policy transitions between subgoals based on the agent’s current inventory. Unlike VPT, we use standard high-level crafting actions that instantly crafts target items from inventory items. At the time of this writing, Minedojo does not support the VPT style of human-like crafting in a GUI, so we instead remove the VPT action for opening the crafting GUI and replace it with 254 crafting actions (one for each item). This brings our multi-discrete action space to 121 camera actions and 8714 keyboard/crafting actions, and our observation space to 128x128x3 pixels plus 391 inventory item quantities (only used to transition between subgoals).
我们在实验中使用 Minedojo 的 Minecraft 实现（Fan 等人，2022）。与 VPT（Baker 等人，2022）一样，我们的子目标策略使用仅像素观察空间和大型多离散动作空间，而我们的整体策略基于代理的当前库存在子目标之间进行转换。与 VPT 不同，我们使用标准的高级制作动作，它可以立即从库存项目中制作目标项目。在撰写本文时，Minedojo 不支持 VPT 风格的类人制作 GUI，因此我们删除了用于打开制作 GUI 的 VPT 动作，并用 254 个制作动作（每个项目一个）替换它。这将我们的多离散动作空间扩展到 121 个相机动作和 8714 个键盘/制作动作，观察空间扩展到 128x128x3 像素加上 391 个库存项目数量（仅用于在子目标之间进行转换）。

Because our subgoals map to individual items, there is an intrinsic separation between items that are collected from the environment versus those that require crafting. While we must finetune a set of adapters for subgoals that require navigating or collecting items from the environment, crafting subgoal policies map to a single craft action—making them much more space and sample efficient compared to collectable item subgoals.
因为我们的子目标映射到单独的物品，所以在收集环境物品与需要制作的物品之间存在固有的区别。虽然我们必须为需要从环境中导航或收集物品的子目标调整一组适配器，但制作子目标策略映射到单个制作操作——这使它们在空间和样本效率方面比可收集物品子目标要高得多。

We evaluate DECKARD on both crafting tasks—in which the agent learns to collect ingredients and craft a target item—and open-ended exploration. In open-ended exploration, although there is no extrinsic learning signal, DECKARD is intrinsically motivated to explore new AWM nodes. We compare the growth of the agent’s verified AWM during open-ended exploration for DECKARD with and without LLM guidance along with a VPT baseline. Next, we compare LLM-guided DECKARD to RL baselines and DECKARD without LLM guidance on goal-driven tasks for collecting/crafting: logs, wooden pickaxes, cobblestone, stone pickaxes, furnaces, sand, and glass. We also compare to several popular Mincraft agents on the “craft a stone pickaxe task” (see Table 2). Finally, we evaluate the effect of artificial errors in the hypothesized AWM to simulate errors in LLM output and demonstrate DECKARD’s robustness to LLM accuracy.
我们在两种制作任务上评估 DECKARD——一种是代理学习收集原料并制作目标物品的任务，另一种是开放式探索任务。在开放式探索中，虽然没有外部学习信号，但 DECKARD 内在地受到探索新 AWM 节点的驱动。我们比较了 DECKARD 在开放式探索中代理的验证 AWM 的增长，分别在有和没有 LLM 引导以及 VPT 基线下进行。接下来，我们将 LLM 引导的 DECKARD 与强化学习基线以及没有 LLM 引导的 DECKARD 在收集/制作目标驱动任务（例如原木、木镐、圆石、石镐、熔炉、沙子和玻璃）上进行比较。我们还将 DECKARD 与几种流行的《我的世界》代理进行比较，针对“制作石镐任务”（见表 2）。最后，我们评估了假设 AWM 中的人工错误对 LLM 输出的影响，并展示了 DECKARD 对 LLM 准确性的稳健性。

DECKARDis intrinsically motivated to explore new nodes, always sampling and attempting to craft new items, and thus does not require a target task to improve exploration. We can measure the effect of DECKARD on exploration by tracking the growth of the agent’s verified AWM nodes. Figure 2 shows the speed of exploration when using DECKARD with and without LLM guidance. We also compare DECKARD to a VPT baseline that explores the environment without an AWM with a non-finetuned VPT policy. Although VPT does not construct an AWM, it gathers Minecraft items and randomly attempts to craft new items from the gathered ingredients. We track how many items it has discovered and plot that quantity in Figure 2. DECKARD without LLM guidance constructs an AWM from scratch, but only the LLM-guided DECKARD agent uses LLM guidance to decide which items to collect and which recipes to attempt next. Note that DECKARD subgoal policies are initialized with VPT, so VPT starts out exploring at a similar rate to DECKARD.
DECKARD天生具有探索新节点的内在动机，始终在采样和尝试制作新物品，因此不需要目标任务来改进探索。我们可以通过跟踪代理程序验证的AWM节点的增长来衡量DECKARD对探索的影响。图2显示了使用DECKARD时探索速度与LLM指导和无指导时的对比。我们还将DECKARD与探索环境时没有AWM的VPT基线进行了比较，后者具有未经优化的VPT策略。虽然VPT不构建AWM，但它会收集Minecraft物品，并随机尝试从收集的成分中制作新物品。我们跟踪它发现了多少物品，并在图2中绘制了该数量。没有LLM指导的DECKARD会从头开始构建AWM，但只有LLM指导的DECKARD代理会使用LLM指导来决定下一步收集哪些物品和尝试哪些配方。请注意，DECKARD子目标策略是用VPT初始化的，因此VPT开始时的探索速度与DECKARD相似。

The DECKARD and VPT agents quickly learn to mine logs and craft wooden items. However, one exploration hurdle is discovering that wooden pickaxes are a prerequisite for mining cobblestone. As seen in Figure 2, it takes DECKARD without LLM guidance and the VPT baseline 2x and 3x longer respectively to learn to use a wooden pickaxe to mine cobblestone. Once the agents learn how to mine cobblestone, they can begin adding stone items to their AWM. However, only DECKARD avoids oversampling dead ends in the crafting tree allowing it to quickly explore new states. Also, the LLM incorrectly predicts that glass can be collected without crafting or tools of any kind, but DECKARD overcomes and corrects this error, successfully crafting glass and adding the correct recipe to the AWM.
DECKARD 和 VPT 代理迅速学会挖掘日志并制作木制物品。然而，一个探索的障碍是发现木镐是挖掘圆石的先决条件。如图2所示，DECKARD 在没有 LLM 指导下学习如何使用木镐挖掘圆石需要的时间分别是 VPT 基线的2倍和3倍长。一旦代理学会了如何挖掘圆石，它们就可以开始将石制品添加到它们的 AWM 中。然而，只有 DECKARD 避免了在制作树中过度采样死胡同，使其能够快速探索新的状态。此外，LLM 错误地预测玻璃可以在不进行任何制作或使用工具的情况下收集，但 DECKARD 克服并纠正了这个错误，成功地制作了玻璃并将正确的配方添加到 AWM 中。

In general, the frontier $F$ of the verified AWM nodes is much smaller than $G$. This difference increases as the agent continues to explore and add verified nodes to $G$. Figure 3 shows the sizes of $G$ and $F$ throughout open-ended exploration for DECKARD. The smaller size of $F$ means that each iteration DECKARD is more likely to sample items that are useful for crafting something new. Eventually, difficult to reach or erroneous nodes in $F$ could limit exploration, so we stop prioritizing sampling from the frontier after $c_{0}$ failed attempts to reach nodes from $F$.
一般来说，已验证的 AWM 节点的边界比较小。随着代理继续探索并向 AWM 添加已验证的节点，这种差异会增加。图 3 显示了 DECKARD 在开放式探索过程中的 $G$ 和 $F$ 的大小。较小的 $F$ 大小意味着每次迭代 DECKARD 更有可能采样到对于创造新物品有用的项目。最终， $F$ 中难以达到或错误的节点可能会限制探索，因此在连续 $c_{0}$ 次尝试未能到达 $F$ 节点后，我们停止优先从边界进行采样。

We also evaluate DECKARD on tasks that challenge the agent to collect or crafting a specific item. The training procedure for items tasks is the same, but rather than sample from the entire frontier $F$ as with open-ended exploration, we only sample nodes from $F$ predicted to lead to the target item. We conclude training when the target item is obtained.
我们还评估 DECKARD 在挑战代理收集或制作特定物品的任务上的表现。物品任务的训练程序与之相同，但不同于从整个边界 $F$ 中采样作为开放式探索，我们仅从预测将导致目标物品的节点 $F$ 中采样。当获得目标物品时，我们结束训练。

Figure 4 compares DECKARD success rates to baselines across item tasks: logs, wooden and stone pickaxes, cobblestone, furnace, sand, and glass. The VPT baseline is the non-finetuned VPT policy acting in the environment, and VPT-a follows the same training setup as our subgoal policies (see Section 5.2). Agents are allowed a maximum of 1,000 environment steps to obtain collectable items (log and sand), and 5,000 steps for all other craftable items. Training for each agent is limited to 6 million steps, although DECKARD only takes that many for the “craft glass” task. DECKARD outperforms directly training on item tasks with a traditional reinforcement learning signal and learns to craft items further up the technology tree where the baseline completely fails.
图 4 对比了 DECKARD 在项目任务中与基线的成功率：原木、木镐和石镐、圆石、熔炉、沙子和玻璃。VPT 基线是在环境中执行的非微调 VPT 策略，VPT-a 遵循与我们的子目标策略相同的训练设置（见第 5.2 节）。代理被允许最多进行 1,000 个环境步骤来获取可收集的物品（原木和沙子），对于所有其他可制作的物品，最多可以进行 5,000 步。每个代理的训练限制为 600 万步，尽管 DECKARD 仅对“制作玻璃”任务进行了相同数量的训练步骤。DECKARD 在直接训练项目任务与传统的强化学习信号相比表现更好，并学会在技术树上进一步制作物品，而基线完全失败。

Note that we use random world seeds for all evaluation making scarce items more difficult to reliably collect. For example, the fact that sand is more rare than logs is reflected in their respective success rates in Figure 4. Also, items that depend on logs (pickaxes, cobblestone, furnace) and sand (glass) will have success rates bounded by that of their parent nodes in the AWM.
请注意，我们在所有评估中使用随机世界种子，使得稀有物品更难以可靠地收集。例如，沙比原木更稀有这一事实在图4中的它们各自的成功率中有所体现。另外，依赖于原木（镐、鹅卵石、熔炉）和沙（玻璃）的物品的成功率将受到其父节点在AWM中的成功率限制。

The sample efficiency of DECKARD with LLM guidance is especially notable when applied to item crafting and collecting tasks. With LLM guidance, DECKARD can avoid learning subgoal policies for items it predicts are unnecessary for the current goal (see Section 4.2.2). Figure 5 demonstrates the large difference in sample efficiency when only training policies for predicted subgoals. Without LLM guidance, DECKARD finetunes subgoal policies for an average of fifteen different collectable items when learning to craft a stone pickaxe. With guidance, DECKARD only finetunes subgoal policies for collecting needed items (such as logs and cobblestone)—resulting in an order of magnitude improvement in sample efficiency.
DECKARD在物品制作和收集任务中的样本效率在有LLM引导时尤其引人注目。有LLM引导时，DECKARD可以避免学习对当前目标预测为不必要的物品的子目标策略（参见第4.2.2节）。图5展示了仅训练预测子目标策略时的样本效率差异。没有LLM引导时，DECKARD在学习制作石镐时，对平均15种不同可收集物品进行子目标策略的微调。有引导时，DECKARD仅对收集所需物品（如原木和圆石）进行子目标策略的微调，样本效率提高了一个数量级。

Although not the primary goal of this work, we compare DECKARD to several agents from previous work trained to craft items along the Minecraft technology tree. Table 2 includes a high level overview of these agents and shows the number of environment samples for each to learn the “craft stone pickaxe” task. Note that each of these agents uses different action and observation spaces as well as pretraining data. For example, DECKARD does not require any reward shaping from domain expertise, expert demonstrations, or simplifications of the observation and action spaces. As mentioned in Section 5.3, we do follow previous work and use discrete actions for item crafting. At the time of writing, VPT is the only agent that learns low-level item crafting using the graphical crafting interface. Table 2 shows that DECKARD’s sample efficiency is equal to or better than that of previous work.
虽然这项工作的主要目标不是这个，但我们将 DECKARD 与先前工作中训练的几个代理进行了比较，这些代理被训练来沿着《我的世界》技术树制作物品。表 2 包括这些代理的高级概述，并显示了每个代理学习“制作石镐”的环境样本数量。请注意，这些代理使用不同的动作和观察空间以及预训练数据。例如，DECKARD 不需要来自领域专业知识、专家演示或观察和行动空间的简化的任何奖励塑造。正如第 5.3 节中所提到的，我们确实遵循先前的工作，并使用离散动作进行物品制作。在撰写本文时，VPT 是唯一使用图形制作界面学习低级物品制作的代理。表 2 显示 DECKARD 的样本效率等于或优于先前的工作。

Finally, we evaluate our claim that DECKARD is robust to errors in LLM output. While LLMs are becoming surprisingly knowledgeable, they are not grounded in environment knowledge and sometimes output erroneous facts (Valmeekam et al., 2022). Figure 6 shows training time for DECKARD on the target task “craft stone pickaxe” for various error types and rates in the hypothesized AWM. For each run, we start with a ground truth AWM and artificially introduce errors over at least three different random seeds for each error type and rate.
最后，我们评估我们关于 DECKARD 对 LLM 输出错误的鲁棒性的声明。虽然 LLMs 变得异常博学，但它们并没有根植于环境知识，有时输出错误的事实（Valmeekam 等人，2022）。图 6 展示了 DECKARD 在目标任务“制作石镐”上的训练时间，针对假设的 AWM 中各种错误类型和错误率。对于每次运行，我们从一个地面真实的 AWM 开始，并在每种错误类型和错误率下的至少三个不同的随机种子上人为引入错误。

The most common error in our LLM-hypothesized AWM was ingredient quantity (see Table 1), but we found that DECKARD was robust to this error and often ended up with a surplus of ingredients. Figure 6 shows the effect of inserting and deleting edges from the ground truth AWM. Inserted edges always add sand as an ingredient for the current item, and deleted edges may remove recipe ingredients or a required tool/crafting table. DECKARD with LLM guidance successfully outperforms DECKARD without LLM guidance even when faced with large errors in LLM output, demonstrating DECKARD’s robustness to LLM output as an exploration method.
在我们的LLM假设的 AWM 中，最常见的错误是成分数量（见表 1），但我们发现 DECKARD 对这种错误具有稳健性，并且通常会产生过剩的成分。图 6 显示了从地面真相 AWM 中插入和删除边缘的效果。插入的边缘总是为当前项目添加沙子作为成分，而删除的边缘可能会删除配方成分或所需的工具/制作桌。具有LLM指导的 DECKARD 成功地优于没有LLM指导的 DECKARD，即使面临LLM输出中的大错误，也证明了 DECKARD 对LLM输出具有探索方法的稳健性。