Deep Q-learning algorithms (Mnih et al., 2015) have been labeled as off-policy due to their connection to off-policy Q-learning (Watkins, 1989). However, these algorithms tend to use near-on-policy exploratory policies, such as $\epsilon$-greedy, in conjunction with a replay buffer (Lin, 1992). As a result, the generated dataset tends to be heavily correlated to the current policy. In this section, we examine how these off-policy algorithms perform when learning with uncorrelated datasets. Our results demonstrate that the performance of a state of the art deep actor-critic algorithm, DDPG (Lillicrap et al., 2015), deteriorates rapidly when the data is uncorrelated and the value estimate produced by the deep Q-network diverges. These results suggest that off-policy deep reinforcement learning algorithms are ineffective when learning truly off-policy.
深度 Q 学习算法（Mnih 等，2015 年）因其与离策略 Q 学习（Watkins，1989 年）的联系而被标记为离策略。然而，这些算法倾向于使用接近在策略的探索性策略，例如ε-贪婪，结合重放缓冲区（Lin，1992 年）。因此，生成的数据集往往与当前策略高度相关。在本节中，我们检查这些离策略算法在使用不相关数据集学习时的表现。我们的结果表明，当数据不相关且深度 Q 网络产生的价值估计发散时，最先进的深度演员-评论家算法 DDPG（Lillicrap 等，2015 年）的性能迅速恶化。这些结果表明，当真正离策略学习时，离策略深度强化学习算法是无效的。

Our practical experiments examine three different batch settings in OpenAI gym’s Hopper-v1 environment (Todorov et al., 2012; Brockman et al., 2016), which we use to train an off-policy DDPG agent with no interaction with the environment. Experiments with additional environments and specific details can be found in the Supplementary Material.
我们的实践实验在 OpenAI gym 的 Hopper-v1 环境中检验了三种不同的批处理设置（Todorov 等人，2012; Brockman 等人，2016），我们使用它来训练一个不与环境交互的离策略 DDPG 代理。关于额外环境的实验和具体细节可以在补充材料中找到。

Batch 1 (Final buffer). We train a DDPG agent for 1 million time steps, adding $\mathcal{N}(0,0.5)$ Gaussian noise to actions for high exploration, and store all experienced transitions. This collection procedure creates a dataset with a diverse set of states and actions, with the aim of sufficient coverage.
批次 1（最终缓冲区）。我们训练一个 DDPG 代理 1 百万时间步长，为了高度探索，在动作中添加 $\mathcal{N}(0,0.5)$ 高斯噪声，并存储所有经历过的转换。这个收集程序创建了一个具有多样化状态和动作的数据集，目的是足够的覆盖率。

Batch 2 (Concurrent). We concurrently train the off-policy and behavioral DDPG agents, for 1 million time steps. To ensure sufficient exploration, a standard $\mathcal{N}(0,0.1)$ Gaussian noise is added to actions taken by the behavioral policy. Each transition experienced by the behavioral policy is stored in a buffer replay, which both agents learn from. As a result, both agents are trained with the identical dataset.
批次 2（并行）。我们同时训练离策略和行为 DDPG 代理，为期 1 百万时间步长。为了确保足够的探索，标准 $\mathcal{N}(0,0.1)$ 高斯噪声被添加到行为策略采取的动作中。行为策略经历的每一个转换都存储在一个缓冲回放中，两个代理都从中学习。结果，两个代理都使用相同的数据集进行训练。

Batch 3 (Imitation). A trained DDPG agent acts as an expert, and is used to collect a dataset of 1 million transitions.
批次 3（模仿）。一个训练有素的 DDPG 代理充当专家，并用来收集 1 百万次转换的数据集。

In Figure 1, we graph the performance of the agents as they train with each batch, as well as their value estimates. Straight lines represent the average return of episodes contained in the batch. Additionally, we graph the learning performance of the behavioral agent for the relevant tasks.
在图 1 中，我们绘制了代理在每个批次训练时的表现以及它们的价值估计。直线代表批次中包含的剧集的平均回报。此外，我们还绘制了行为代理在相关任务上的学习表现。

Our experiments demonstrate several surprising facts about off-policy deep reinforcement learning agents. In each task, the off-policy agent performances significantly worse than the behavioral agent. Even in the concurrent experiment, where both agents are trained with the same dataset, there is a large gap in performance in every single trial. This result suggests that differences in the state distribution under the initial policies is enough for extrapolation error to drastically offset the performance of the off-policy agent. Additionally, the corresponding value estimate exhibits divergent behavior, while the value estimate of the behavioral agent is highly stable. In the final buffer experiment, the off-policy agent is provided with a large and diverse dataset, with the aim of providing sufficient coverage of the initial policy. Even in this instance, the value estimate is highly unstable, and the performance suffers. In the imitation setting, the agent is provided with expert data. However, the agent quickly learns to take non-expert actions, under the guise of optimistic extrapolation. As a result, the value estimates rapidly diverge and the agent fails to learn.
我们的实验展示了关于离策略深度强化学习代理的几个令人惊讶的事实。在每项任务中，离策略代理的表现显著低于行为代理。即使在同时进行的实验中，当两个代理使用相同的数据集进行训练时，每一次试验中都存在很大的性能差距。这一结果表明，初始策略下状态分布的差异足以使外推误差极大地影响离策略代理的性能。此外，相应的价值估计表现出发散行为，而行为代理的价值估计非常稳定。在最终的缓冲实验中，离策略代理被提供了一个大型且多样化的数据集，旨在提供初始策略的足够覆盖。即便在这种情况下，价值估计也高度不稳定，性能也受到影响。在模仿设置中，代理被提供了专家数据。然而，代理很快学会采取非专家动作，这是在乐观外推的幌子下进行的。结果，价值估计迅速发散，代理未能学习。

Although extrapolation error is not necessarily positively biased, when combined with maximization in reinforcement learning algorithms, extrapolation error provides a source of noise that can induce a persistent overestimation bias (Thrun & Schwartz, 1993; Van Hasselt et al., 2016; Fujimoto et al., 2018). In an on-policy setting, extrapolation error may be a source of beneficial exploration through an implicit “optimism in the face of uncertainty” strategy (Lai & Robbins, 1985; Jaksch et al., 2010). In this case, if the value function overestimates an unknown state-action pair, the policy will collect data in the region of uncertainty, and the value estimate will be corrected. However, when learning off-policy, or in a batch setting, extrapolation error will never be corrected due to the inability to collect new data.
尽管外推误差不一定是正向偏差的，但当与强化学习算法中的最大化结合时，外推误差提供了一种噪声源，这种噪声源可以诱导持续的过高估计偏差（Thrun & Schwartz, 1993; Van Hasselt et al., 2016; Fujimoto et al., 2018）。在一个按策略的设置中，外推误差可能是通过一种隐含的“面对不确定性时的乐观”策略来进行有益探索的来源（Lai & Robbins, 1985; Jaksch et al., 2010）。在这种情况下，如果价值函数对未知的状态-动作对进行了过高估计，策略将在不确定性区域收集数据，价值估计将被纠正。然而，在离策略学习或批量设置中，由于无法收集新数据，外推误差将永远不会被纠正。

These experiments show extrapolation error can be highly detrimental to learning off-policy in a batch reinforcement learning setting. While the continuous state space and multi-dimensional action space in MuJoCo environments are contributing factors to extrapolation error, the scale of these tasks is small compared to real world settings. As a result, even with a sufficient amount of data collection, extrapolation error may still occur due to the concern of catastrophic forgetting (McCloskey & Cohen, 1989; Goodfellow et al., 2013). Consequently, off-policy reinforcement learning algorithms used in the real-world will require practical guarantees without exhaustive amounts of data.
这些实验表明，在批量强化学习设置中，外推误差可能对离策略学习极为不利。虽然 MuJoCo 环境中的连续状态空间和多维动作空间是外推误差的促成因素，但这些任务的规模与现实世界的设置相比还是小的。因此，即使有足够多的数据收集，由于灾难性遗忘的问题（McCloskey & Cohen, 1989; Goodfellow et al., 2013），外推误差仍可能发生。因此，在现实世界中使用的离策略强化学习算法将需要在没有大量数据的情况下提供实际保证。

In the finite MDP setting, extrapolation error can be described by the bias from the mismatch between the transitions contained in the buffer and the true MDP. We find that by inducing a data distribution that is contained entirely within the batch, batch-constrained policies can eliminate extrapolation error entirely for deterministic MDPs. In addition, we show that the batch-constrained variant of Q-learning converges to the optimal policy under the same conditions as the standard form of Q-learning. Moreover, we prove that for a deterministic MDP, batch-constrained Q-learning is guaranteed to match, or outperform, the behavioral policy when starting from any state contained in the batch. All of the proofs for this section can be found in the Supplementary Material.
在有限 MDP 设置中，外推误差可以通过缓冲区中包含的转换与真实 MDP 之间的偏差来描述。我们发现，通过引入完全包含在批次中的数据分布，批量约束策略可以完全消除确定性 MDP 的外推误差。此外，我们展示了批量约束变体的 Q 学习在与标准形式的 Q 学习相同的条件下收敛到最优策略。此外，我们证明了对于一个确定性 MDP，批量约束 Q 学习保证能够匹配或超越批次中包含的任何状态的行为策略。本节的所有证明都可以在补充材料中找到。

A value estimate $Q$ can be learned using an experience replay buffer $\mathcal{B}$. This involves sampling transition tuples $(s,a,r,s^{\prime})$ with uniform probability, and applying the temporal difference update (Sutton, 1988; Watkins, 1989):
可以使用经验回放缓冲区 $\mathcal{B}$ 学习价值估计 $Q$ 。这涉及到以均匀概率抽样转换元组 $(s,a,r,s^{\prime})$ ，并应用时序差分更新（Sutton, 1988; Watkins, 1989）：

If $\pi(s^{\prime})=\operatorname*{argmax}_{a^{\prime}}Q(s^{\prime},a^{\prime})$, this is known as Q-learning. Assuming a non-zero probability of sampling any possible transition tuple from the buffer and infinite updates, Q-learning converges to the optimal value function.
如果 $\pi(s^{\prime})=\operatorname*{argmax}_{a^{\prime}}Q(s^{\prime},a^{\prime})$ ，这被称为 Q 学习。假设从缓冲区抽样任何可能的转换元组的非零概率和无限次更新，Q 学习收敛到最优价值函数。

We begin by showing that the value function $Q$ learned with the batch $\mathcal{B}$ corresponds to the value function for an alternate MDP $M_{\mathcal{B}}$. From the true MDP $M$ and initial values $Q(s,a)$, we define the new MDP $M_{\mathcal{B}}$ with the same action and state space as $M$, along with an additional terminal state $s_{\text{init}}$. $M_{\mathcal{B}}$ has transition probabilities $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$, where $N(s,a,s^{\prime})$ is the number of times the tuple $(s,a,s^{\prime})$ is observed in $\mathcal{B}$. If $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$, then $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$, where $r(s,a,s_{\text{init}})$ is set to the initialized value of $Q(s,a)$.
我们首先展示用批次 $\mathcal{B}$ 学习到的价值函数 $Q$ 对应于另一个 MDP $M_{\mathcal{B}}$ 的价值函数。从真实的 MDP $M$ 和初始值 $Q(s,a)$ 出发，我们定义了新的 MDP $M_{\mathcal{B}}$ ，它与 $M$ 有相同的动作和状态空间，并增加了一个额外的终止状态 $s_{\text{init}}$ 。 $M_{\mathcal{B}}$ 的转移概率为 $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$ ，其中 $N(s,a,s^{\prime})$ 是在 $\mathcal{B}$ 中观察到元组 $(s,a,s^{\prime})$ 的次数。如果 $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$ ，那么 $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$ ，其中 $r(s,a,s_{\text{init}})$ 被设置为 $Q(s,a)$ 的初始值。

Theorem 1. Performing Q-learning by sampling from a batch $\mathcal{B}$ converges to the optimal value function under the MDP $M_{\mathcal{B}}$.
定理 1：通过从批次 $\mathcal{B}$ 中采样执行 Q 学习，在 MDP $M_{\mathcal{B}}$ 下收敛到最优值函数。

We define $\epsilon_{\text{MDP}}$ as the tabular extrapolation error, which accounts for the discrepancy between the value function $Q^{\pi}_{\mathcal{B}}$ computed with the batch $\mathcal{B}$ and the value function $Q^{\pi}$ computed with the true MDP $M$:
我们将 $\epsilon_{\text{MDP}}$ 定义为表格外推误差，它解释了用批次 $\mathcal{B}$ 计算的值函数 $Q^{\pi}_{\mathcal{B}}$ 与用真实 MDP $M$ 计算的值函数 $Q^{\pi}$ 之间的差异：

For any policy $\pi$, the exact form of $\epsilon_{\text{MDP}}(s,a)$ can be computed through a Bellman-like equation:
对于任何策略 $\pi$ ， $\epsilon_{\text{MDP}}(s,a)$ 的确切形式可以通过类似贝尔曼的方程计算出来：

This means extrapolation error is a function of divergence in the transition distributions, weighted by value, along with the error at succeeding states. If the policy is chosen carefully, the error between value functions can be minimized by visiting regions where the transition distributions are similar. For simplicity, we denote
这意味着外推误差是转移分布差异的函数，按价值加权，连同在后续状态的误差。如果策略选择得当，通过访问转移分布相似的区域，可以最小化价值函数之间的误差。为了简单起见，我们表示

To evaluate a policy $\pi$ exactly at relevant state-action pairs, only $\epsilon^{\pi}_{\text{MDP}}=0$ is required. We can then determine the condition required to evaluate the exact expected return of a policy without extrapolation error.
要准确评估策略 $\pi$ 在相关状态-动作对上，只需要 $\epsilon^{\pi}_{\text{MDP}}=0$ 。然后我们可以确定评估策略的确切预期回报而无需外推误差的所需条件。

Lemma 1. For all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$ for all $s^{\prime}\in\mathcal{S}$ and $(s,a)$ such that $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$.
引理 1：对于所有奖励函数，当且仅当对于所有的 $s^{\prime}\in\mathcal{S}$ 和 $(s,a)$ ，满足 $\mu_{\pi}(s)>0$ 和 $\pi(a|s)>0$ 时， $\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当 $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$ 。

Lemma 1 states that if $M_{\mathcal{B}}$ and $M$ exhibit the same transition probabilities in regions of relevance, the policy can be accurately evaluated. For a stochastic MDP this may require an infinite number of samples to converge to the true distribution, however, for a deterministic MDP this requires only a single transition. This means a policy which only traverses transitions contained in the batch, can be evaluated without error. More formally, we denote a policy $\pi\in\Pi_{\mathcal{B}}$ as batch-constrained if for all $(s,a)$ where $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$ then $(s,a)\in\mathcal{B}$. Additionally, we define a batch $\mathcal{B}$ as coherent if for all $(s,a,s^{\prime})\in\mathcal{B}$ then $s^{\prime}\in\mathcal{B}$ unless $s^{\prime}$ is a terminal state. This condition is trivially satisfied if the data is collected in trajectories, or if all possible states are contained in the batch. With a coherent batch, we can guarantee the existence of a batch-constrained policy.
引理 1 表明，如果 $M_{\mathcal{B}}$ 和 $M$ 在相关区域展示出相同的转移概率，那么策略可以被准确评估。对于随机的 MDP，这可能需要无限多的样本才能收敛到真实分布，然而，对于确定性的 MDP，这只需要一个单一转移。这意味着，一个仅遍历批次中包含的转移的策略，可以无误差地被评估。更正式地，我们将一个策略 $\pi\in\Pi_{\mathcal{B}}$ 定义为批次约束的，如果对于所有 $(s,a)$ ，满足 $\mu_{\pi}(s)>0$ 和 $\pi(a|s)>0$ ，则 $(s,a)\in\mathcal{B}$ 。此外，我们定义一个批次 $\mathcal{B}$ 为连贯的，如果对于所有 $(s,a,s^{\prime})\in\mathcal{B}$ ，则 $s^{\prime}\in\mathcal{B}$ ，除非 $s^{\prime}$ 是一个终止状态。如果数据是按轨迹收集的，或者批次包含了所有可能的状态，那么这个条件就会被轻易满足。有了一个连贯的批次，我们可以保证存在一个批次约束的策略。

Theorem 2. For a deterministic MDP and all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if the policy $\pi$ is batch-constrained. Furthermore, if $\mathcal{B}$ is coherent, then such a policy must exist if the start state $s_{0}\in\mathcal{B}$.
定理 2：对于一个确定性的 MDP 和所有奖励函数，当且仅当策略 $\pi$ 是批量约束的时， $\epsilon^{\pi}_{\text{MDP}}=0$ 成立。此外，如果 $\mathcal{B}$ 是连贯的，那么如果起始状态 $s_{0}\in\mathcal{B}$ ，则必须存在这样的策略。

Batch-constrained policies can be used in conjunction with Q-learning to form batch-constrained Q-learning (BCQL), which follows the standard tabular Q-learning update while constraining the possible actions with respect to the batch:
批量约束策略可以与 Q 学习结合使用，形成批量约束 Q 学习（BCQL），它遵循标准的表格 Q 学习更新，同时根据批量约束可能的行动：

BCQL converges under the same conditions as the standard form of Q-learning, noting the batch-constraint is nonrestrictive given infinite state-action visitation.
在与标准 Q 学习相同的条件下，BCQL 收敛，注意到在无限状态-动作访问下，批量约束是非限制性的。

Theorem 3. Given the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$, and standard sampling requirements from the environment, BCQL converges to the optimal value function $Q^{*}$.
定理 3：给定学习率 $\alpha$ 上的 Robbins-Monro 随机收敛条件，以及来自环境的标准采样要求，BCQL 收敛到最优值函数 $Q^{*}$ 。

The more interesting property of BCQL is that for a deterministic MDP and any coherent batch $\mathcal{B}$, BCQL converges to the optimal batch-constrained policy $\pi^{*}\in\Pi_{\mathcal{B}}$ such that $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$ for all $\pi\in\Pi_{\mathcal{B}}$ and $(s,a)\in\mathcal{B}$.
BCQL 的一个更有趣的特性是，对于一个确定性的 MDP 和任何一致的批次 $\mathcal{B}$ ，BCQL 会收敛到最优的批次约束策略 $\pi^{*}\in\Pi_{\mathcal{B}}$ ，使得对所有 $\pi\in\Pi_{\mathcal{B}}$ 和 $(s,a)\in\mathcal{B}$ 。

Theorem 4. Given a deterministic MDP and coherent batch $\mathcal{B}$, along with the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$ and standard sampling requirements on the batch $\mathcal{B}$, BCQL converges to $Q^{\pi}_{\mathcal{B}}(s,a)$ where $\pi^{*}(s)=\operatorname*{argmax}_{a\text{ s.t.}(s,a)\in\mathcal{B}}Q^{\pi}_{\mathcal{B}}(s,a)$ is the optimal batch-constrained policy.
定理 4。给定一个确定性的 MDP 和一致的批次 $\mathcal{B}$ ，以及对学习率 $\alpha$ 的 Robbins-Monro 随机收敛条件和对批次 $\mathcal{B}$ 的标准采样要求，BCQL 收敛到 $Q^{\pi}_{\mathcal{B}}(s,a)$ ，其中 $\pi^{*}(s)=\operatorname*{argmax}_{a\text{ s.t.}(s,a)\in\mathcal{B}}Q^{\pi}_{\mathcal{B}}(s,a)$ 是最优的批次约束策略。

This means that BCQL is guaranteed to outperform any behavioral policy when starting from any state contained in the batch, effectively outperforming imitation learning. Unlike standard Q-learning, there is no condition on state-action visitation, other than coherency in the batch.
这意味着 BCQL 保证在从批次中包含的任何状态开始时，都会胜过任何行为策略，有效地胜过模仿学习。与标准的 Q 学习不同，除了批次的一致性外，没有对状态-动作访问的条件。

We introduce our approach to off-policy batch reinforcement learning, Batch-Constrained deep Q-learning (BCQ). BCQ approaches the notion of batch-constrained through a generative model. For a given state, BCQ generates plausible candidate actions with high similarity to the batch, and then selects the highest valued action through a learned Q-network. Furthermore, we bias this value estimate to penalize rare, or unseen, states through a modification to Clipped Double Q-learning (Fujimoto et al., 2018). As a result, BCQ learns a policy with a similar state-action visitation to the data in the batch, as inspired by the theoretical benefits of its tabular counterpart.
我们介绍了我们对离线批量强化学习的方法，批量约束深度 Q 学习（BCQ）。BCQ 通过一个生成模型来处理批量约束的概念。对于给定的状态，BCQ 生成与批量高度相似的合理候选动作，然后通过学习到的 Q 网络选择最高价值的动作。此外，我们通过对剪切双 Q 学习（Fujimoto 等，2018）的修改，对这一价值估计进行偏差，以惩罚罕见或未见的状态。因此，受其表格式对应物的理论优势的启发，BCQ 学习了一种与批量数据中的状态-动作访问相似的策略。

To maintain the notion of batch-constraint, we define a similarity metric by making the assumption that for a given state $s$, the similarity between $(s,a)$ and the state-action pairs in the batch $\mathcal{B}$ can be modelled using a learned state-conditioned marginal likelihood $P_{\mathcal{B}}^{G}(a|s)$. In this case, it follows that the policy maximizing $P_{\mathcal{B}}^{G}(a|s)$ would minimize the error induced by extrapolation from distant, or unseen, state-action pairs, by only selecting the most likely actions in the batch with respect to a given state. Given the difficulty of estimating $P_{\mathcal{B}}^{G}(a|s)$ in high-dimensional continuous spaces, we instead train a parametric generative model of the batch $G_{\omega}(s)$, which we can sample actions from, as a reasonable approximation to $\operatorname*{argmax}_{a}P_{\mathcal{B}}^{G}(a|s)$.
为了维持批次约束的概念，我们通过假设对于给定状态 $s$ ， $(s,a)$ 与批次 $\mathcal{B}$ 中的状态-动作对之间的相似性可以使用学习到的状态条件边缘似然 $P_{\mathcal{B}}^{G}(a|s)$ 来建模来定义一个相似性度量。在这种情况下，可以推断出，最大化 $P_{\mathcal{B}}^{G}(a|s)$ 的策略将通过仅选择批次中相对于给定状态最可能的动作，来最小化由远距离或未见状态-动作对引起的外推误差。鉴于在高维连续空间中估计 $P_{\mathcal{B}}^{G}(a|s)$ 的难度，我们改为训练批次 $G_{\omega}(s)$ 的参数化生成模型，我们可以从中采样动作，作为对 $\operatorname*{argmax}_{a}P_{\mathcal{B}}^{G}(a|s)$ 的合理近似。

For our generative model we use a conditional variational auto-encoder (VAE) (Kingma & Welling, 2013; Sohn et al., 2015), which models the distribution by transforming an underlying latent space¹¹1See the Supplementary Material for an introduction to VAEs.. The generative model $G_{\omega}$, alongside the value function $Q_{\theta}$, can be used as a policy by sampling $n$ actions from $G_{\omega}$ and selecting the highest valued action according to the value estimate $Q_{\theta}$. To increase the diversity of seen actions, we introduce a perturbation model $\xi_{\phi}(s,a,\Phi)$, which outputs an adjustment to an action $a$ in the range $[-\Phi,\Phi]$. This enables access to actions in a constrained region, without having to sample from the generative model a prohibitive number of times. This results in the policy $\pi$:
对于我们的生成模型，我们使用了一个条件变分自编码器（VAE）（Kingma & Welling, 2013; Sohn et al., 2015），它通过转换一个潜在空间 ¹ 来建模分布。生成模型 $G_{\omega}$ ，连同价值函数 $Q_{\theta}$ ，可以通过从 $G_{\omega}$ 采样 $n$ 动作并根据价值估计 $Q_{\theta}$ 选择最高价值动作作为策略。为了增加所见动作的多样性，我们引入了一个扰动模型 $\xi_{\phi}(s,a,\Phi)$ ，它输出一个在范围 $[-\Phi,\Phi]$ 内对动作 $a$ 的调整。这使得在不需要从生成模型中抽样过多次的情况下，访问一个受限区域内的动作成为可能。这导致了策略 $\pi$ ：

The choice of $n$ and $\Phi$ creates a trade-off between an imitation learning and reinforcement learning algorithm. If $\Phi=0$, and the number of sampled actions $n=1$, then the policy resembles behavioral cloning and as $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ and $n\rightarrow\infty$, then the algorithm approaches Q-learning, as the policy begins to greedily maximize the value function over the entire action space.
$n$ 和 $\Phi$ 的选择在模仿学习和强化学习算法之间创造了一种权衡。如果 $\Phi=0$ ，且采样动作的数量 $n=1$ ，那么策略类似于行为克隆；随着 $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ 和 $n\rightarrow\infty$ ，算法则趋近于 Q 学习，因为策略开始贪婪地在整个动作空间上最大化价值函数。