Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks
开放世界长期任务的技能强化学习和规划

In Minecraft, a task $\tau=(g,I)$ is defined with the combination of a goal $g$ and the agent’s initial condition $I$, where $g$ represents the target entity to acquire in the task and $I$ represents the initial tools and conditions provided for the agent. For example, a task can be ‘harvest cooked_beef  with sword  in plains’. We model the task as a partially observable Markov decision process (POMDP) [16]. $I$ determines the environment’s initial state distribution. At each timestep $t$, the agent obtains the partial observation $o_{t}$, takes an action $a_{t}$ following its policy $\pi(a_{t}|o_{0:t},\tau)$, and receives a sparse reward $r_{t}$ indicating task completion. The agent aims to maximize its expected return $R=\mathbb{E}_{\pi}\sum_{t}\gamma^{t}r_{t}$.
在 Minecraft 中，任务 $\tau=(g,I)$ 由目标 $g$ 和代理的初始条件 $I$ 的组合定义，其中 $g$ 表示任务中要获取的目标实体， $I$ 表示为代理提供的初始工具和条件。例如，一个任务可以是“在平原上用剑 收获熟牛肉 ”。我们将任务建模为部分可观察的马尔可夫决策过程（POMDP）[16]。 $I$ 确定了环境的初始状态分布。在每个时间步骤 $t$ ，代理获取部分观察 $o_{t}$ ，根据其策略 $\pi(a_{t}|o_{0:t},\tau)$ 采取行动 $a_{t}$ ，并接收表示任务完成的稀疏奖励 $r_{t}$ 。代理的目标是最大化其预期回报 $R=\mathbb{E}_{\pi}\sum_{t}\gamma^{t}r_{t}$ 。

To solve complex tasks, humans acquire and reuse skills in the world, rather than learn each task independently from scratch. Similarly, to solve the aforementioned task, the agent can sequentially use the skills: harvest log , …, craft furnace , harvest beef , place furnace , and craft cooked_beef . Each skill solves a simple sub-task in a shorter time horizon, with the necessary tools and conditions provided. For example, the skill ‘craft cooked_beef ’ solves the task ‘harvest cooked_beef  with beef , log , and placed furnace ’. Once the agent acquires an abundant set of skills $S$, it can solve any complex task by decomposing it into a sequence of sub-tasks and executing the skills in order. Meanwhile, by reusing a skill to solve different tasks, the agent is much better in memory and learning efficiency.
为了解决复杂的任务，人类在世界中获取和重复使用技能，而不是从头开始独立学习每个任务。同样，为了解决上述任务，代理可以按顺序使用以下技能：收获木材 ，…，制作熔炉 ，收获牛肉 ，放置熔炉 ，制作熟牛肉 。每个技能在较短的时间范围内解决一个简单的子任务，并提供必要的工具和条件。例如，技能“制作熟牛肉 ”解决了任务“使用牛肉 、木材 和放置的熔炉 收获熟牛肉 ”。一旦代理获得了丰富的技能集合 $S$ ，它可以通过将复杂任务分解为一系列子任务并按顺序执行技能来解决任何复杂任务。同时，通过重复使用技能来解决不同的任务，代理在记忆和学习效率方面更加出色。

To this end, we convert the goal of solving diverse and long-horizon tasks in Minecraft into building a hierarchical agent. At the low level, we train policies $\pi_{s}$ to learn all the skills $s\in S$, where $\pi_{s}$ takes as input the RGB image and some auxiliary information (compass, location, biome, etc.), then outputs an action. At the high level, we study planning methods to convert a task $\tau$ into a skill sequence $(s_{\tau,1},s_{\tau,2},\cdots)$.
为了实现这一目标，我们将解决 Minecraft 中各种长期任务的目标转化为构建一个分层代理。在低层次上，我们训练策略 $\pi_{s}$ 来学习所有的技能 $s\in S$ ，其中 $\pi_{s}$ 以 RGB 图像和一些辅助信息（指南针、位置、生物群系等）作为输入，然后输出一个动作。在高层次上，我们研究规划方法将一个任务 $\tau$ 转化为一个技能序列 $(s_{\tau,1},s_{\tau,2},\cdots)$ 。

Recent works mainly rely on imitation learning to learn Minecraft skills efficiently. In MineRL competition [17], a human gameplay dataset is accessible along with the Minecraft environment. All of the top methods in competition use imitation learning to some degree, to learn useful behaviors in limited interactions. In VPT [2], a large policy model is pre-trained on a massive labeled dataset using behavior cloning. By fine-tuning on smaller datasets, policies are acquired for diverse skills.
最近的研究主要依赖于模仿学习来高效地学习 Minecraft 技能。在 MineRL 比赛[17]中，提供了一个人类游戏数据集以及 Minecraft 环境。所有在比赛中排名靠前的方法都在一定程度上使用了模仿学习，以在有限的交互中学习有用的行为。在 VPT[2]中，使用行为克隆在大规模标记数据集上预训练了一个大型策略模型。通过在较小的数据集上微调，可以获得各种技能的策略。

However, without demonstration datasets, learning Minecraft skills with reinforcement learning (RL) is difficult. MineAgent [9] shows that PPO [32] can only learn a small set of skills. PPO with sparse reward fails in ‘milk a cow’ and ‘shear a sheep’, though the distance between target mobs and the agent is set within 10 blocks. We argue that with the high dimensional state and action space, open-ended large world, and partial observation, exploration in Minecraft tasks is extremely difficult.
然而，没有演示数据集，使用强化学习（RL）学习Minecraft技能是困难的。MineAgent [9]表明，PPO [32]只能学习一小部分技能。尽管目标生物和代理之间的距离设置在10个方块内，但PPO在“挤奶牛”和“剪羊毛”任务中的稀疏奖励失败。我们认为，由于高维状态和动作空间、开放式的大世界和部分观察，Minecraft任务中的探索极其困难。

We conduct a study for RL to learn skills with different difficulties in Table 1. We observe that RL has comparable performance to imitation learning only when the task-relevant entities are initialized very close to the agent. Otherwise, RL performance decreases significantly. This motivates us to further divide skills into fine-grained skills. We propose a Finding-skill to provide a good initialization for other skills. For example, the skill of ‘milk a cow’ is decomposed into ‘find a cow’ and ‘harvest milk_bucket’. After finding a cow nearby, ‘harvest milk_bucket’ can be accomplished by RL with acceptable sample efficiency. Thus, learning such fine-grained skills is easier for RL, and they together can still accomplish the original task.
我们在表 1 中进行了一个强化学习的研究，以学习具有不同难度的技能。我们观察到，只有当与任务相关的实体初始化非常接近代理时，强化学习的性能才与模仿学习相当。否则，强化学习的性能会显著下降。这促使我们进一步将技能细分为细粒度的技能。我们提出了一种寻找技能，为其他技能提供良好的初始化。例如，‘挤奶’这个技能被分解为‘找到一头奶牛’和‘收获牛奶桶’。在附近找到一头奶牛后，可以通过强化学习以可接受的样本效率完成‘收获牛奶桶’。因此，对于强化学习来说，学习这些细粒度的技能更容易，并且它们仍然可以完成原始任务。

Finding items is a long-horizon difficult task for RL. To find an unseen tree on the plains, the agent should take thousands of steps to explore the world map as much as possible. A random policy fails to do such exploration, as shown in Appendix A. Also, it is too costly to train different policies for various target items. To simplify this problem, considering to explore on the world’s surface only, we propose to train a target-free hierarchical policy to solve all the Finding-skills.
对于强化学习来说，寻找物品是一项长期困难的任务。为了在平原上找到一棵未知的树，代理人需要花费数千步来尽可能地探索世界地图。随机策略无法进行这样的探索，如附录 A 所示。此外，为不同的目标物品训练不同的策略成本太高。为了简化这个问题，我们提出了在世界表面上进行探索的无目标分层策略，以解决所有的寻找技能。

Figure 2 demonstrates the hierarchical policy for Finding-skills. The high-level policy $\pi^{H}\left((x,y)^{g}|(x,y)_{0:t}\right)$ observes historical locations $(x,y)_{0:t}$ of the agent, and outputs a goal location $(x,y)^{g}$. It drives the low-level policy $\pi^{L}\left(a_{t}|o_{t},(x,y)^{g}\right)$ to reach the goal location. We assume that target items are uniformly distributed on the world’s surface. To maximize the chance to find diverse targets, the objective for the high-level policy is to maximize its reached area. We divide the world’s surface into discrete grids, where each grid represents a $10\times 10$ area. We use state count in the grids as the reward for the high-level policy. The low-level policy obtains the environmental observation $o_{t}$ and the goal location $(x,y)^{g}$ proposed by the high-level policy, and outputs an action $a_{t}$. We reward the low-level policy with the distance change to the goal location.
图2展示了寻找技能的层次策略。高层策略观察代理的历史位置，并输出一个目标位置。它驱动低层策略达到目标位置。我们假设目标物品均匀分布在世界表面上。为了最大化找到多样化目标的机会，高层策略的目标是最大化其到达的区域。我们将世界表面划分为离散的网格，每个网格代表一个区域。我们使用网格中的状态计数作为高层策略的奖励。低层策略获取环境观察和高层策略提出的目标位置，并输出一个动作。我们根据到目标位置的距离变化奖励低层策略。

To train the hierarchical policy with acceptable sample complexity, we pre-train the low-level policy with randomly generated goal locations using DQN [26], then train the high-level policy using PPO [32] with the fixed low-level policy. During test, to find a specific item, the agent first explores the world with the hierarchical policy until a target item is detected in its lidar observations. Then, the agent executes the low-level policy conditioned on the detected target’s location, to reach the target item. Though we use additional lidar information here, we believe that without this information, we can also implement the success detector for Finding-skills with computer vision models [7].
为了以可接受的样本复杂度训练分层策略，我们使用DQN [26]对低层策略进行预训练，使用固定的低层策略使用PPO [32]训练高层策略。在测试过程中，为了找到特定的物品，智能体首先使用分层策略在世界中进行探索，直到在其激光雷达观测中检测到目标物品。然后，智能体根据检测到的目标位置执行低层策略，以达到目标物品。尽管我们在这里使用了额外的激光雷达信息，但我们相信即使没有这些信息，我们也可以使用计算机视觉模型实现找到技能的成功检测器[7]。

By executing the pre-trained Finding-skills, we can instantiate the manipulation tasks with requisite target items nearby, making the manipulation tasks much easier. To train the Manipulation-skills in Minecraft, we can either make a training environment with the target item initialized nearby or run the Finding-skills to reach a target item. For example, to train the skill ‘harvest milk_bucket ’, we can either spawn a cow  close to the agent using the Minecraft built-in commands, or execute the Finding-skills until a cow is reached. The latter is similar in the idea to Go-Explore [8], and is more suitable for other environments that do not have commands to initialize the target items nearby.
通过执行预训练的寻找技能，我们可以在附近实例化具有所需目标物品的操作任务，使操作任务变得更容易。要在 Minecraft 中训练操作技能，我们可以创建一个训练环境，并在附近初始化目标物品，或者运行寻找技能以达到目标物品。例如，要训练技能“收获牛奶桶 ”，我们可以使用 Minecraft 内置命令在代理附近生成一头牛 ，或者执行寻找技能直到找到一头牛。后者与 Go-Explore [8]的思想类似，并且更适用于没有命令在附近初始化目标物品的其他环境。

We adopt MineCLIP [9] to guide the agent with intrinsic rewards. The pre-trained MineCLIP model computes the CLIP reward based on the similarity between environmental observations (frames) and the language descriptions of the skill. We train the agent using PPO with self-imitation learning, to maximize a weighted sum of intrinsic rewards and extrinsic success (sparse) reward. Details for training basic skills can be found in Appendix D.
我们采用 MineCLIP [9]来通过内在奖励指导代理。预训练的 MineCLIP 模型根据环境观察（帧）与技能的语言描述之间的相似性计算 CLIP 奖励。我们使用自我模仿学习的 PPO 来训练代理，以最大化内在奖励和外在成功（稀疏）奖励的加权和。有关训练基本技能的详细信息，请参见附录 D。

For the Crafting-skills, they can be executed with only a single action in MineDojo [9].
对于制作技能，它们可以在 MineDojo 中仅通过一个动作来执行。

A correct plan $(s_{\tau,1},s_{\tau,2},\cdots)$ for a task $\tau=(g,I)$ should satisfy two conditions. (1) For each $i$, $s_{\tau,i}$ is executable after $(s_{\tau,1},\cdots,s_{\tau,i-1})$ are accomplished sequentially with initial condition $I$. (2) The target item $g$ is obtained after all the skills are accomplished sequentially, given initial condition $I$. To enable searching for such plans, we should be able to verify whether a plan is correct. Thus, we should know what condition is required and what is obtained for each skill. We define such information of skills in a structured format. As an example, information for skill ‘crafting stone_pickaxe ’ is:
一个任务的正确计划应满足两个条件。(1) 对于每个，当顺序完成初始条件后，可执行。(2) 在所有技能按顺序完成后，目标项目被获得，给定初始条件。为了能够搜索这样的计划，我们应该能够验证计划是否正确。因此，我们应该知道每个技能所需的条件和获得的条件。我们以结构化格式定义这些技能的信息。例如，技能“制作石镐”的信息如下：

Each item in this format is also a skill. Regarding them as graph nodes, this format shows a graph structure between skill ‘stone_pickaxe’ and skills ‘cobblestone’, ‘stick’, ‘crafting_table_nearby’. The directed edge from ‘cobblestone‘ to ‘stone_pickaxe’ is represented as (3, 1, consume), showing the quantity relationship between parent and child, and that the parent item will be consumed during skill execution. In fact, in this format, all the basic skills in Minecraft construct a large directed acyclic graph with hundreds of nodes. The dashed box in Figure 1 shows a small part of this graph, where grey arrows denote ‘consume’ and red arrows denote ‘require’.
每个以这种格式表示的物品也是一种技能。将它们视为图节点，这种格式显示了技能“stone_pickaxe”与技能“cobblestone”、“stick”、“crafting_table_nearby”之间的图结构。从“cobblestone”到“stone_pickaxe”的有向边表示为（3, 1, consume），显示了父项和子项之间的数量关系，并且在技能执行过程中将消耗父项物品。实际上，在这种格式中，Minecraft 中的所有基本技能构成了一个具有数百个节点的大型有向无环图。图 1 中的虚线框显示了该图的一小部分，其中灰色箭头表示“consume”，红色箭头表示“require”。

To construct the skill graph, we generate structured information for all the skills by interacting with ChatGPT (GPT-3.5) [29], a high-performance LLM. Since LLMs are trained on large-scale internet datasets, they obtain rich knowledge in the popular game Minecraft. In prompt, we give a few demonstrations and explanations about the format, then ask ChatGPT to generate other skills information. Dialog with ChatGPT can be found in Appendix E.
为了构建技能图，我们通过与 ChatGPT（GPT-3.5）[29]进行交互，为所有技能生成结构化信息。由于LLM在大规模的互联网数据集上进行训练，它们在流行游戏 Minecraft 中获得了丰富的知识。在提示中，我们给出了一些示例和关于格式的解释，然后要求 ChatGPT 生成其他技能信息。与 ChatGPT 的对话可以在附录 E 中找到。

Our skill planning method is a depth-first search (DFS) algorithm on the skill graph. Given a task $\tau=(g,I)$, we start from the node $g$ and do DFS toward its parents, opposite to the edge directions. In this process, we maintain all the possessing items starting from $I$. Once conditions for the skill are satisfied or the skill node has no parent, we append this skill into the planned skill list and modify the maintained items according to the skill information. The resulting skill list is ensured to be executable and target-reaching.
我们的技能规划方法是在技能图上进行深度优先搜索（DFS）算法。给定一个任务 $\tau=(g,I)$ ，我们从节点 $g$ 开始，沿着反向边向其父节点进行 DFS。在这个过程中，我们维护从 $I$ 开始的所有拥有的物品。一旦技能的条件满足或者技能节点没有父节点，我们将这个技能添加到计划的技能列表中，并根据技能信息修改维护的物品。确保生成的技能列表可执行且达到目标。

To solve a long-horizon task, since the learned low-level skills are possible to fail, we alternate skill planning and skill execution until the episode terminates. After each skill execution, we update the agent’s condition $I^{\prime}$ based on its inventory and the last executed skill, and search for the next skill with $\tau^{\prime}=(g,I^{\prime})$.
为了解决一个长期任务，由于学习到的低级技能可能会失败，我们在每次技能执行之后，根据代理的库存和上次执行的技能更新代理的状态 $I^{\prime}$ ，并搜索下一个具有 $\tau^{\prime}=(g,I^{\prime})$ 的技能，直到该情节终止。

We present the pseudocode for the skill search algorithm and the testing process in Appendix C.
我们在附录 C 中提供了技能搜索算法和测试过程的伪代码。

To pre-train basic skills with RL, we use the environments of programmatic tasks in MineDojo [9]. To train Manipulation-skills, for simplicity, we specify the environment that initializes target mobs or resources close to the agent. For the Go-Explore-like training method without specified environments discussed in Section 3.2, we present the results in Appendix H, which does not underperform the former.
为了使用强化学习预训练基本技能，我们使用了 MineDojo [9]中的编程任务环境。为了训练操作技能，为了简单起见，我们指定了一个环境，该环境在代理附近初始化目标生物或资源。关于没有指定环境的类似 Go-Explore 的训练方法的结果，请参见附录 H，该方法不会表现不佳。

For Manipulation-skills and the low-level policy of Finding-skills, we adopt the policy architecture of MineAgent [9], which uses a fixed pre-trained MineCLIP image encoder and processes features using MLPs. To explore in a compact action space, we compress the original large action space into $12\times 3$ discrete actions. For the high-level policy of Finding-skills, which observes the agent’s past locations, we use an LSTM policy and train it with truncated BPTT [30]. We pick the model with the highest success rate on the smoothed training curve for each skill, and fix these policies in all tasks. Implementation details can be found in Appendix D.
对于操作技能和低级策略的寻找技能，我们采用了MineAgent [9]的策略架构，该架构使用了一个固定的预训练MineCLIP图像编码器，并使用MLP处理特征。为了在紧凑的动作空间中进行探索，我们将原始的大动作空间压缩为离散动作。对于高级策略的寻找技能，它观察到代理的过去位置，我们使用了一个LSTM策略，并使用截断的BPTT [30]进行训练。我们选择在平滑的训练曲线上具有最高成功率的模型，并将这些策略固定在所有任务中。实现细节可以在附录D中找到。

Note that Plan4MC totally takes 7M environmental steps in training, and can unlock the iron pickaxe  in the Minecraft Tech Tree in test. The sample efficiency greatly outperforms all other existing demonstration-free RL methods [12, 2].
请注意，Plan4MC 在训练中完全采取了 7M 个环境步骤，并且可以在测试中解锁 Minecraft 技术树中的铁镐 。样本效率远远超过所有其他现有的无演示强化学习方法[12, 2]。

Based on MineDojo [9] programmatic tasks, we set up an evaluation benchmark consisting of four groups of diverse tasks: cutting trees  to craft primary items, mining cobblestones  to craft intermediate items, mining iron ores  to craft advanced items, and interacting with mobs  to harvest food and materials. Each task set has 10 tasks, adding up to a total of 40 tasks. With our settings of basic skills, these tasks require 25 planning steps on average and maximally 121 planning steps. We estimate the number of the required steps for each task with the sum of the steps of the initially planned skills and double this number to be the maximum episode length for the task, allowing skill executions to fail. The easiest tasks have 3000 maximum steps, while the hardest tasks have 12000. More details about task setup are listed in Appendix F. To evaluate the success rate on each task, we average the results over 30 test episodes.
基于MineDojo [9]的编程任务，我们设置了一个评估基准，包括四组不同的任务：砍树 以制作初级物品，挖掘鹅卵石 以制作中级物品，挖掘铁矿石 以制作高级物品，与生物互动 以收获食物和材料。每组任务有10个任务，总共40个任务。根据我们设置的基本技能，这些任务平均需要25个规划步骤，最多121个规划步骤。我们估计每个任务所需的步骤数，是初始计划技能步骤的总和，并将此数字加倍作为任务的最大回合数，允许技能执行失败。最简单的任务最多有3000个步骤，而最困难的任务最多有12000个步骤。有关任务设置的更多详细信息请参见附录F。为了评估每个任务的成功率，我们对30个测试回合的结果进行平均。

We first analyze learning basic skills. While we propose three types of fine-grained basic skills, others directly learn more complicated and long-horizon skills. We introduce two baselines to study learning skills with RL.
我们首先分析学习基本技能。虽然我们提出了三种细粒度的基本技能，但其他人直接学习更复杂和长期的技能。我们引入了两个基准来研究使用强化学习学习技能。

MineAgent [9]. Without decomposing tasks into basic skills, MineAgent solves tasks using PPO and self-imitation learning with the CLIP reward. For fairness, we train MineAgent in the test environment for each task. The training takes 7M environmental steps, which is equal to the sum of environmental steps we take for training all the basic skills. We average the success rate of trajectories in the last 100 training epochs (around 1M environment steps) to be its test success rate. Since MineAgent has no actions for crafting items, we hardcode the crafting actions into the training code. During trajectory collection, at each time step where the skill search algorithm returns a Crafting-skill, the corresponding crafting action will be executed. Note that, if we expand the action space for MineAgent rather than automatically execute crafting actions, the exploration will be much harder.
MineAgent [9]。在不将任务分解为基本技能的情况下，MineAgent 使用 PPO 和自我模仿学习以 CLIP 奖励解决任务。为了公平起见，我们为每个任务在测试环境中训练 MineAgent。训练共进行了 700 万个环境步骤，这相当于我们用于训练所有基本技能的环境步骤总和。我们将最后 100 个训练时期（约 100 万个环境步骤）的轨迹成功率平均为其测试成功率。由于 MineAgent 没有用于制作物品的动作，我们将制作动作硬编码到训练代码中。在轨迹收集过程中，每当技能搜索算法返回一个制作技能时，相应的制作动作将被执行。请注意，如果我们扩展 MineAgent 的动作空间而不是自动执行制作动作，探索将变得更加困难。

Plan4MC w/o Find-skill. None of the previous work decomposes a skill into executing Finding-skills and Manipulation-skills. Instead, finding items and manipulations are done with a single skill. Plan4MC w/o Find-skill implements such a method. It skips all the Finding-skills in the skill plans during test. Manipulation-skills take over the whole process of finding items and manipulating them.
Plan4MC 无查找技能。以往的工作都没有将技能分解为执行查找技能和操作技能。相反，查找物品和操作都是使用单一技能完成的。Plan4MC 无查找技能实现了这种方法。在测试过程中，它跳过了技能计划中的所有查找技能。操作技能接管了查找物品和操作的整个过程。

Table 2 shows the test results for these methods. Plan4MC outperforms two baselines on the four task sets. MineAgent fails on the task sets of Cut-Trees, Mine-Stones and Mine-Ores, since taking many attacking actions continually to mine blocks in Minecraft is an exploration difficulty for RL on long-horizon tasks. On the contrary, MineAgent achieves performance comparable to Plan4MC’s on some easier tasks  in Interact-Mobs, which requires fewer environmental steps and planning steps. Plan4MC w/o Find-skill consistently underperforms Plan4MC on all the tasks, showing that introducing Finding-skills is beneficial for solving hard tasks with basic skills trained by RL. Because there is no Finding-skill in harvesting iron ores, their performance gap on Mine-Ores tasks is small.
表2显示了这些方法的测试结果。Plan4MC在四个任务集上表现优于两个基准方法。MineAgent在Cut-Trees、Mine-Stones和Mine-Ores任务集上失败，因为在Minecraft中连续采矿需要进行许多攻击动作，这对于长期任务上的强化学习来说是一个探索困难。相反，MineAgent在Interact-Mobs中的一些较简单的任务上表现与Plan4MC相当，这些任务需要较少的环境步骤和规划步骤。Plan4MC在所有任务上没有Find-skill的情况下始终表现不佳，表明引入Finding-skills对于通过强化学习训练的基本技能解决困难任务是有益的。因为在采集铁矿石的任务中没有Find-skill，所以它们在Mine-Ores任务上的表现差距很小。

To further study Finding-skills, we present the success rate at each planning step in Figure 3 for three tasks. The curves of Plan4MC and Plan4MC w/o Find-skill have large drops at Finding-skills. Especially, the success rates at finding cobblestones and logs decrease the most, because these items are harder to find in the environment compared to mobs. In these tasks, we compute the average success rate of Manipulation-Skills, conditioned on the skill before the last Finding-skills being accomplished. While Plan4MC has a conditional success rate of 0.40, Plan4MC w/o Find-skill decreases to 0.25, showing that solving sub-tasks with additional Finding-skills is more effective.
进一步研究寻找技能，我们在图3中呈现了三个任务中每个规划步骤的成功率。Plan4MC和Plan4MC w/o Find-skill的曲线在寻找技能阶段有大幅下降。特别是，在找到鹅卵石和原木方面的成功率下降最多，因为与敌对生物相比，在环境中找到这些物品更加困难。在这些任务中，我们计算了Manipulation-Skills的平均成功率，条件是在最后一个寻找技能之前的技能已经完成。虽然Plan4MC的条件成功率为0.40，但Plan4MC w/o Find-skill降至0.25，表明使用额外的寻找技能来解决子任务更加有效。

As shown in Table 3, most Manipulation-skills have slightly lower success rates in test than in training, due to the domain gap between test and training environments. However, this decrease does not occur in skills  that are trained with a large initial distance of mobs/items, as pre-executed Finding-skills provide better initialization for Manipulation-skills during the test and thus the success rate may increase. In contrast, the success rates in the test without Finding-skills are significantly lower.
如表3所示，大多数操作技能在测试中的成功率略低于训练中的成功率，这是由于测试和训练环境之间的领域差异所致。然而，对于那些在训练时与移动物体/物品之间有较大初始距离的技能，这种下降并不会发生，因为预先执行的寻找技能在测试期间为操作技能提供了更好的初始化，从而可能提高成功率。相比之下，没有寻找技能的测试成功率显著较低。

For skill planning in open-ended worlds, recent works [13, 14, 3, 21, 37] generate plans or sub-tasks with LLMs. We study these methods on our task sets and implement a best-performing baseline to compare with Plan4MC.
对于开放式世界中的技能规划，最近的研究[13, 14, 3, 21, 37]使用LLMs生成计划或子任务。我们在我们的任务集上研究了这些方法，并实现了一个表现最佳的基准线与 Plan4MC 进行比较。

Interactive LLM. We implement an interactive planning baseline using LLMs. We take ChatGPT [29] as the planner, which proposes skill plans based on prompts including descriptions of tasks and observations. Similar to chain-of-thoughts prompting [38], we provide few-shot demonstrations with explanations to the planner at the initial planning step. In addition, we add several rules for planning into the prompt to fix common errors that the model encountered during test. At each subsequent planning step, the planner will encounter one of the following cases: the proposed skill name is invalid, the skill is already done, skill execution succeeds, and skill execution fails. We carefully design language feedback for each case and ask the planner to re-plan based on inventory changes. For low-level skills, we use the same pre-trained skills as Plan4MC.
交互式LLM。我们使用LLMs实现了一个交互式规划基准线。我们将 ChatGPT [29]作为规划器，它根据任务描述和观察提出技能计划。类似于思维链提示[38]，我们在初始规划步骤中向规划器提供了带有解释的少样本演示。此外，我们在提示中添加了一些规则来修复模型在测试过程中遇到的常见错误。在每个后续的规划步骤中，规划器将遇到以下情况之一：提议的技能名称无效，技能已经完成，技能执行成功，技能执行失败。我们为每种情况精心设计了语言反馈，并要求规划器根据库存变化重新规划。对于低级技能，我们使用与 Plan4MC 相同的预训练技能。

Also, we conduct ablations on our skill planning designs.
此外，我们对我们的技能规划设计进行了消融实验。

Plan4MC Zero-shot. This is a zero-shot variant of our interactive planning method, proposing a skill sequence at the beginning of each task only. The agent executes the planned skills sequentially until a skill fails or the environment terminates. This planner has no fault tolerance for skills execution.
Plan4MC 零射击。这是我们交互式规划方法的零射击变体，仅在每个任务开始时提出一个技能序列。代理按顺序执行计划的技能，直到某个技能失败或环境终止。该规划器对技能执行没有容错能力。

Plan4MC 1/2-steps. In this ablation study, we half the test episode length and require the agent to solve tasks more efficiently.
Plan4MC 1/2 步。在这个消融研究中，我们将测试剧集长度减半，并要求代理更高效地解决任务。

Success rates for each method are listed in Table 2. We find that Interactive LLM has comparable performance to Plan4MC on the task set of Interact-Mobs, where most tasks require less than 10 planning steps. In Mine-Stones and Mine-Ores tasks with long-horizon planning, the LLM planner is more likely to make mistakes, resulting in worse performance. The performance of Plan4MC Zero-shot is much worse than Plan4MC in all the tasks, since a success test episode requires accomplishing each skill in one trial. The decrease is related to the number of planning steps and skills success rates in Table 3. Plan4MC 1/2-steps has the least performance decrease to Plan4MC, showing that Plan4MC can solve tasks in a more limited episode length.
每种方法的成功率在表 2 中列出。我们发现，在 Interact-Mobs 任务集中，大多数任务需要少于 10 个规划步骤，交互式LLM的性能与 Plan4MC 相当。在需要长期规划的 Mine-Stones 和 Mine-Ores 任务中，LLM规划器更容易出错，导致性能较差。在所有任务中，Plan4MC 零-shot 的性能要比 Plan4MC 差得多，因为一个成功测试剧集需要在一次试验中完成每个技能。这种下降与表 3 中的规划步骤数量和技能成功率有关。Plan4MC 1/2-steps 对 Plan4MC 的性能下降最小，表明 Plan4MC 可以在更有限的剧集长度内解决任务。