What’s Behind PPO’s Collapse in Long-CoT?
Value Optimization Holds the Secret
PPO 在长 CoT 中的崩溃背后是什么？价值优化掌握着秘密

Reinforcement Learning (RL) centers around the learning of a policy that maximizes the cumulative reward for an agent as it interacts with an environment. In this study, we cast language generation tasks within the framework of a Markov Decision Process (MDP) [12].
强化学习（RL）的核心是学习一个策略，该策略在智能体与环境交互时最大化累积奖励。在本研究中，我们将语言生成任务纳入马尔可夫决策过程（MDP）[12]的框架内。

Let the prompt be denoted as $x$, and the response to this prompt as $y$. Both $x$ and $y$ can be decomposed into sequences of tokens. For example, the prompt $x$ can be expressed as $x=(x_{0},\dots,x_{m})$, where the tokens are drawn from a fixed discrete vocabulary $\mathcal{A}$.
提示符表示为 $x$ ，对此提示符的响应表示为 $y$ 。 $x$ 和 $y$ 都可以分解为标记序列。例如，提示符 $x$ 可以表示为 $x=(x_{0},\dots,x_{m})$ ，其中标记来自一个固定的离散词汇表 $\mathcal{A}$ 。

We define the token-level MDP as the tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},r,d_{0},\omega)$. Here is a detailed breakdown of each component:
我们将标记级 MDP 定义为元组 $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},r,d_{0},\omega)$ 。以下是每个组件的详细分解：

• •

  State Space ($\mathcal{S}$): This space encompasses all possible states formed by the tokens generated up to a given time step. At time step $t$, the state $s_{t}$ is defined as $s_{t}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t})$.

  Report issue for preceding element
  
  状态空间（ $\mathcal{S}$ ）：该空间包含在给定时间步之前由标记生成的所有可能状态。在时间步 $t$ ，状态 $s_{t}$ 被定义为 $s_{t}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t})$ 。
• •

  Action Space ($\mathcal{A}$): It corresponds to the fixed discrete vocabulary, from which tokens are selected during the generation process.

  Report issue for preceding element
  
  动作空间（ $\mathcal{A}$ ）：对应于固定的离散词汇表，在生成过程中从中选择标记。
• •

  Dynamics ($\mathbb{P}$): These represent a deterministic transition model between tokens. Given a state $s_{t}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t})$, an action $a=y_{t+1}$, and the subsequent state $s_{t+1}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t},y_{t+1})$, the probability $\mathbb{P}(s_{t+1}|s_{t},a)=1$.

  Report issue for preceding element
  
  • 动力学（ $\mathbb{P}$ ）：这代表了一种在标记之间的确定性转换模型。给定一个状态 $s_{t}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t})$ ，一个动作 $a=y_{t+1}$ ，以及后续状态 $s_{t+1}=(x_{0},\dots,x_{m},y_{0},\dots,y_{t},y_{t+1})$ ，概率 $\mathbb{P}(s_{t+1}|s_{t},a)=1$ 。
• •

  Termination Condition: The language generation process concludes when the terminal action $\omega$, typically the end-of-sentence token, is executed.

  Report issue for preceding element
  
  终止条件：当执行终端动作 $\omega$ ，通常是句末标记时，语言生成过程结束。
• •

  Reward Function ($r(s,a)$): This function offers scalar feedback to evaluate the agent’s performance after taking action $a$ in state $s$. In the context of Reinforcement Learning from Human Feedback (RLHF), the reward function can be learned from human preferences or defined by a set of rules specific to the task.

  Report issue for preceding element
  
  奖励函数（ $r(s,a)$ ）：该函数在动作 $a$ 后，在状态 $s$ 下提供标量反馈以评估智能体的性能。在人类反馈强化学习（RLHF）的背景下，奖励函数可以从人类偏好中学习或由特定于任务的规则集定义。
• •

  Initial State Distribution ($d_{0}$): It is a probability distribution over prompts $x$. An initial state $s_{0}$ consists of the tokens within the prompt $x$.

  Report issue for preceding element
  
  初始状态分布（ $d_{0}$ ）：它是关于提示词 $x$ 的概率分布。一个初始状态 $s_{0}$ 由提示词 $x$ 中的标记组成。

We formulate the optimization problem as a KL-regularized RL task. Our objective is to approximate the optimal KL-regularized policy, which is given by:
我们将优化问题表述为 KL 正则化的强化学习任务。我们的目标是逼近最优的 KL 正则化策略，其表达式为：

In this equation, $H$ represents the total number of decision steps, $s_{0}$ is a prompt sampled from the dataset, $r(s_{h},a_{h})$ is the token-level reward obtained from the reward function, $\beta$ is a coefficient that controls the strength of the KL-regularization, and $\pi_{\text{ref}}$ is the initialization policy.
在这个方程中， $H$ 代表决策步骤总数， $s_{0}$ 是从数据集中采样的提示， $r(s_{h},a_{h})$ 是从奖励函数获得的 token 级奖励， $\beta$ 是控制 KL 正则化强度的系数， $\pi_{\text{ref}}$ 是初始化策略。

In traditional RLHF and most tasks related to Large Language Models (LLMs), the reward is sparse and is only assigned at the terminal action $\omega$, that is, the end-of-sentence token <eos>.
在传统的 RLHF 以及与大型语言模型(LLMs)相关的多数任务中，奖励是稀疏的，并且仅在终端动作( $\omega$ )时分配，即句尾标记。

PPO [16] uses a clipped surrogate objective to update the policy. The key idea is to limit the change in the policy during each update step, preventing large policy updates that could lead to instability.
PPO [16] 使用剪枝代理目标来更新策略。关键思想是在每个更新步骤中限制策略的变化，防止可能导致不稳定的较大策略更新。

Let $\pi_{\theta}(a|s)$ be the policy parameterized by $\theta$, and $\pi_{\theta_{\text{old}}}(a|s)$ be the old policy from the previous iteration. The surrogate objective function for PPO is defined as:
令 $\pi_{\theta}(a|s)$ 为由 $\theta$ 参数化的策略， $\pi_{\theta_{\text{old}}}(a|s)$ 为前一次迭代的旧策略。PPO 的代理目标函数定义为：

where $r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{t})}$ is the probability ratio, $\hat{A}_{t}$ is the estimated advantage at time step $t$, and $\epsilon$ is a hyperparameter that controls the clipping range.
$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{t})}$ 是概率比， $\hat{A}_{t}$ 是在时间步 $t$ 估计的优势， $\epsilon$ 是控制裁剪范围的超参数。

Generalized Advantage Estimation (GAE) [15] is a technique used to estimate the advantage function more accurately in PPO. It combines multiple-step bootstrapping to reduce the variance of the advantage estimates. For a trajectory of length $T$, the advantage estimate $\hat{A}_{t}$ at time step $t$ is computed as:
广义优势估计（GAE）[15]是一种用于在 PPO 中更精确估计优势函数的技术。它结合多步自举来减少优势估计的方差。对于长度为 $T$ 的轨迹，时间步 $t$ 的优势估计 $\hat{A}_{t}$ 计算如下：

where $\gamma$ is the discount factor, $\lambda\in[0,1]$ is the GAE parameter, and $\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})$ is the temporal-difference (TD) error. Here, $r_{t}$ is the reward at time step $t$, and $V(s)$ is the value function. Since it is a common practice to use discount factor $\gamma=1.0$ in RLHF, to simplify our notation, we omit $\gamma$ in later sections of this paper.
$\gamma$ 是折扣因子， $\lambda\in[0,1]$ 是 GAE 参数， $\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})$ 是时序差分（TD）误差。在这里， $r_{t}$ 是时间步 $t$ 的奖励， $V(s)$ 是价值函数。由于在强化学习与人类反馈（RLHF）中常用折扣因子 $\gamma=1.0$ ，为了简化符号，本文后续部分省略 $\gamma$ 。

Two common practices when applying PPO in the domain of Reinforcement Learning from Human Feedback (RLHF) are as follows [22, 7]:
两种在人类反馈强化学习（RLHF）领域应用 PPO 的常见做法如下[22, 7]：

• •

  Employ the default Generalized Advantage Estimation (GAE), typically with $\lambda=0.95$.

  Report issue for preceding element
  
  • 使用默认的广义优势估计（GAE），通常为 $\lambda=0.95$ 。
• •

  Initialize the value model using a well-trained reward model.

  Report issue for preceding element
  
  初始化价值模型，使用经过良好训练的奖励模型。

The first practice finds its origin in the traditional RL literature, where PPO has been extensively tested in environments like Mujoco [19] and Atari [2]. In these environments, the rewards accumulate over the trajectory, resulting in high-variance return. As a consequence, variance reduction becomes a necessity. The second practice emerges naturally from the apparent similarity between a reward model and a value model, since both models are trained to predict scalar information about the response. However, our experiments have revealed that naively applying PPO to tasks that require long CoT inevitably leads to failure, as shown in Figure 1.
第一次实践源于传统的强化学习文献，其中 PPO 在 Mujoco[19]和 Atari[2]等环境中得到了广泛测试。在这些环境中，奖励在轨迹上累积，导致高方差回报。因此，方差减少成为必要。第二次实践自然地从奖励模型和价值模型之间的明显相似性中产生，因为这两个模型都是训练来预测关于响应的标量信息。然而，我们的实验表明，天真地将 PPO 应用于需要长 CoT 的任务必然会导致失败，如图 1 所示。

Typically, the failure modes are that the validation performance degrades as the training starts, accompanied by a significant decrease in the model’s output length. Since it has been demonstrated that output length is strongly correlated with the model’s performance on complex reasoning tasks [10], the initial collapse in output length can be attributed as the root cause of this performance degradation.
通常，失败模式是随着训练开始，验证性能下降，伴随着模型输出长度的显著减少。由于已经证明输出长度与模型在复杂推理任务上的性能密切相关[10]，因此输出长度的初始崩溃可以归因为此性能下降的根本原因。

In our tasks, a verifier serves as the source of the reward signal. It utilizes a rule-based answer parsing mechanism, which is unlikely to show a preference for output length. Consequently, the reduction in output length can only be ascribed to the policy optimization dynamics, which are mainly driven by the advantages assigned to each token. To further explore this, we plot the correlation between advantages and the position of tokens, as shown in Figure 2. This reveals a strong correlation between advantages and token position. The more preceding the tokens are, the more positively biased their advantages are. This causes the model to favor preceding tokens, ultimately leading to the observed collapse in output length.
在我们的任务中，验证器作为奖励信号的来源。它使用基于规则的答案解析机制，不太可能表现出对输出长度的偏好。因此，输出长度的减少只能归因于策略优化动态，这些动态主要是由分配给每个标记的优势驱动的。为了进一步探索这一点，我们绘制了优势和标记位置之间的相关性，如图 2 所示。这揭示了优势和标记位置之间存在着强烈的关联。标记越靠前，它们的优势就越偏向积极。这导致模型倾向于优先考虑前面的标记，最终导致观察到的输出长度减少。

The root cause of the positive bias lies in the objective mismatch between the reward model and the value model. The training objective of the reward model is to score the response at the <EOS> token. Since the tokens preceding <EOS> are not included in the training, the reward model tends to assign lower scores to tokens that are more preceding due to its increasing incompleteness. On the other hand, the aim of value prediction is to estimate the expected rewards of each token preceding <EOS> for a given policy. Given that tokens which are more preceding have lower scores and the KL-penalties are essentially zero at the beginning of training, there will be a positive bias at every timestep $t$ that accumulates along the trajectory, which is obvious through how advantages $\hat{A}_{t}$ is computed:
积极偏差的根本原因在于奖励模型和价值模型之间的客观不匹配。奖励模型的训练目标是给标记的响应评分。由于在训练中不包括之前的标记，因此奖励模型倾向于给更早出现的标记分配较低的分数，因为它的不完整性不断增加。另一方面，价值预测的目标是估计给定策略下每个之前标记的预期奖励。鉴于更早出现的标记分数较低，且在训练初期 KL 惩罚本质上为零，每个时间步 $t$ 都会出现正偏差，并沿着轨迹累积，这通过如何计算优势 $\hat{A}_{t}$ 是显而易见的：

This explains why tokens that are preceding tend to exhibit a greater positive bias in advantages. Due to this correlation between token position and advantages, the model tends to generate shorter responses, which prevents the model from generating a long chain of thoughts before finalizing an answer.
这解释了为什么位于前面的标记往往在优势方面表现出更大的积极偏差。由于标记位置和优势之间的这种相关性，模型倾向于生成较短的响应，这防止了模型在最终确定答案之前生成长链思维。

To alleviate such value initialization bias, we introduce Value-Pretraining. This approach involves offline training the value model until convergence under a pre-specified fixed policy. Once the value model has converged, it will be employed in all subsequent formal experiments. The specific steps are outlined as follows:
为了缓解此类值初始化偏差，我们引入了值预训练。这种方法涉及在预指定的固定策略下离线训练值模型，直到收敛。一旦值模型收敛，它将被用于所有后续的正式实验。具体步骤如下：

1. 1.

   Continuously generate responses by sampling from a fixed policy, for instance, $\pi_{\text{sft}}$, and update the value model using GAE with $\lambda=1.0$, also known as the Monte-Carlo return. This setting transforms the optimization problem into a stable gradient-descent optimization, ensuring more reliable and consistent updates to the value model.

   Report issue for preceding element
   
   1. 通过从固定策略中采样连续生成响应，例如 $\pi_{\text{sft}}$ ，并使用 GAE（ $\lambda=1.0$ ，也称为蒙特卡洛回报）更新价值模型。此设置将优化问题转化为稳定的梯度下降优化，确保对价值模型的更新更加可靠和一致。
2. 2.

   Train the value model until key training metrics, including value loss and explained variance [5], attain sufficiently low values. Monitoring these metrics is crucial as they reflect the quality and stability of the model’s learning process, and reaching low values indicates that the model is converging effectively.

   Report issue for preceding element
   
   2. 训练价值模型，直到关键训练指标，包括价值损失和解释方差[5]，达到足够低的值。监控这些指标至关重要，因为它们反映了模型学习过程的质量和稳定性，达到低值表明模型正在有效收敛。
3. 3.

   Save the value checkpoint upon the completion of training. Subsequently, load this checkpoint for following experiments. This step provides a more accurate initial point for value estimation, enabling the model to start from a well-calibrated state.

   Report issue for preceding element
   
   3. 在训练完成后保存值检查点。随后，为后续实验加载此检查点。这一步骤为价值估计提供了一个更准确的初始点，使模型能够从一个校准良好的状态开始。

Variance reduction is a critical topic in RL. The use of GAE with $\lambda=0.95$ is common in traditional RL tasks like Mujoco and Atari, where accumulated rewards have high variance and lead to slow convergence. In contrast, in RLHF, a reward model or rule-based scoring mechanism offers trajectory-level feedback which consists of non-accumulating and well-defined values.
方差减少是强化学习中的一个关键主题。在传统的强化学习任务中，如 Mujoco 和 Atari，使用 GAE 与 $\lambda=0.95$ 是常见的，因为累积奖励具有高方差，导致收敛速度慢。相比之下，在 RLHF 中，一个奖励模型或基于规则的评分机制提供轨迹级别的反馈，该反馈由非累积且定义良好的值组成。

Therefore, we question whether variance reduction is necessary in optimizing the value model in RLHF.
因此，我们质疑在强化学习与人类反馈（RLHF）中优化价值模型时是否需要方差减少。

Based on the GAE computation in Equation 4, we can rewrite the equation to obtain the value optimization target $V^{\text{target}}$:
基于方程 4 中的 GAE 计算，我们可以将方程重写为获得值优化目标 $V^{\text{target}}$ ：

According to Equation 5, the reward assigned at the <EOS> token decays at a rate of $\lambda$ when propagating to the preceding tokens during GAE computation. The reward signal propagated to the $t$-th token is $\lambda^{T-t}r_{\text{<EOS>}}$. When $T-t$ is large, the resulting reward signal is essentially zero. With $\lambda=1.0$, such reward signal decay would not occur, which makes it a desirable option for value optimization. Moreover, when $\lambda<1.0$, value prediction is incorporated into the construction of the regression target. This approach belongs to semi-gradient descent methods, which tend to be unstable. Conversely, when $\lambda=1.0$, the value is simply regressing to the accumulated rewards, resulting in a stable gradient-descent optimization.
根据方程 5，在 GAE 计算过程中，分配给标记的奖励在传播到前面的标记时以 $\lambda$ 的速度衰减。传播到 $t$ 标记的奖励信号是 $\lambda^{T-t}r_{\text{<EOS>}}$ 。当 $T-t$ 很大时，产生的奖励信号基本上为零。使用 $\lambda=1.0$ ，这种奖励信号衰减不会发生，这使得它成为价值优化的理想选择。此外，当 $\lambda<1.0$ 时，将价值预测纳入回归目标的构建。这种方法属于半梯度下降法，通常是不稳定的。相反，当 $\lambda=1.0$ 时，价值简单地回归到累积奖励，从而实现稳定的梯度下降优化。

In Figure 3, we show that with $\lambda<1.0$, the reward signal rapidly decays during propagation and preceding tokens are unable to receive the signal from the reward model. This phenomenon is exacerbated in tasks that require long CoT because the trajectory lengths are substantially longer. Therefore, optimizing the value in an unbiased manner outweighs learning it in a variance-reduced way because of the trajectory-level reward signal in RLHF. A similar argument is also proposed in [1].
在图 3 中，我们展示了在 $\lambda<1.0$ 的情况下，奖励信号在传播过程中迅速衰减，先前的标记无法接收到奖励模型的信号。这种现象在需要长 CoT 的任务中加剧，因为轨迹长度显著更长。因此，由于 RLHF 中的轨迹级奖励信号，以无偏方式优化价值比以方差减少的方式学习它更为重要。类似的论点也在[1]中提出。

However, variance reduction might still be necessary in policy optimization.
然而，在策略优化中可能仍然需要减少方差。

Training large language models consumes a vast amount of computational resources. Under the constraints of time and computing power, achieving faster convergence during training is highly desirable. In PPO, the $\lambda$ parameter in GAE plays a crucial role in the bias-variance trade-off during policy updates. The variance of policy update can be analyzed in terms of the variances of the TD errors. Let $\text{Var}[\delta_{t}]$ denote the variance of the TD error at time step $t$. The variance of $A_{t}^{\lambda}$ can be roughly computed as:
训练大型语言模型消耗大量计算资源。在时间和计算能力的限制下，在训练过程中实现更快收敛是非常理想的。在 PPO 中，GAE 中的 $\lambda$ 参数在策略更新过程中的偏差-方差权衡中起着关键作用。策略更新的方差可以通过 TD 误差的方差来分析。令 $\text{Var}[\delta_{t}]$ 表示时间步 $t$ 的 TD 误差的方差。 $A_{t}^{\lambda}$ 的方差可以大致计算如下：

where $\text{Cov}[\delta_{t+i},\delta_{t+j}]$ is the covariance between the TD errors at time steps $t+i$ and $t+j$. Since $\lambda\in[0,1]$, as $\lambda$ decreases, the weights $\lambda^{l}$ for the later TD errors decrease more rapidly. This means that the contribution of the more variable and less reliable later TD errors to the overall advantage estimate is reduced, thereby reducing the variance of the advantage estimate.
$\text{Cov}[\delta_{t+i},\delta_{t+j}]$ 是时间步 $t+i$ 和 $t+j$ 之间 TD 错误的协方差。由于 $\lambda\in[0,1]$ ，随着 $\lambda$ 的减小，后续 TD 错误的权重 $\lambda^{l}$ 减少得更快。这意味着更易变且可靠性较低的后续 TD 错误对整体优势估计的贡献减少，从而降低了优势估计的方差。

Nevertheless, adjusting this $\lambda$ can have an additional impact on value optimization. To address this issue, we introduce Decoupled-GAE. This approach allows the policy to adopt a different $\lambda$ value from that of the value function. By doing so, the policy can better balance its own bias-variance trade-off, thereby enhancing the training efficiency.
尽管如此，调整此 $\lambda$ 可能会对价值优化产生额外影响。为了解决这个问题，我们引入了解耦-GAE。这种方法允许策略采用与价值函数不同的 $\lambda$ 值。通过这样做，策略可以更好地平衡其自身的偏差-方差权衡，从而提高训练效率。

Next, we show that using a value function obtained with a different $\lambda$ from the policy is mathematically justifiable without introducing additional bias. Let $\bar{V}$ represent the value estimate obtained with a potentially different $\lambda$, and define the $n$-step return with $\bar{V}$ as $G$:
接下来，我们表明使用与策略不同的 $\lambda$ 获得的价值函数在数学上是合理的，而不会引入额外的偏差。用 $\bar{V}$ 表示可能不同的 $\lambda$ 获得的价值估计，并定义以 $\bar{V}$ 为 $G$ 的 $n$ 步回报：

Then, the policy gradient with an arbitrary $\lambda$ can be rewritten as follows:
然后，可以将具有任意 $\lambda$ 的政策梯度重写如下：

Based on Equation 8, plugging in an arbitrary value function does not introduce additional bias to the policy gradient. Given the substantial time and computational resources required for large language models, it is desirable to use a smaller $\lambda$ to expedite the convergence of the policy. A potential configuration could be $\lambda_{\text{policy}}=0.95$ and $\lambda_{\text{value}}=1.0$.
基于方程 8，将任意值函数代入不会引入额外的偏差到策略梯度。鉴于大型语言模型所需的大量时间和计算资源，使用较小的 $\lambda$ 以加速策略的收敛是可取的。一种可能的配置是 $\lambda_{\text{policy}}=0.95$ 和 $\lambda_{\text{value}}=1.0$ 。

By combining Value-Pretraining and Decoupled-GAE, we propose Value-Calibrated Proximal Policy Optimization (VC-PPO) as shown in Alg 1, which is a simple yet effective approach to enhance PPO’s performance in long CoT tasks. The main differences between VC-PPO and baseline PPO are highlighted.
通过结合价值预训练和解耦 GAE，我们提出了价值校准近端策略优化（VC-PPO），如算法 1 所示，这是一种简单而有效的方法，用于增强长 CoT 任务中 PPO 的性能。VC-PPO 与基线 PPO 之间的主要区别被突出显示。

Datasets. To comprehensively demonstrate the effectiveness of our proposed algorithm, we conduct experiments on American Invitational Mathematics Examination (AIME) problems, which typically demand a long chain of thought for solution. The test set consists of AIME questions from the most recent two years. Conversely, the training set is composed of questions from all past AIME competitions, supplemented with some artificially constructed difficult mathematical problems. To evaluate the model’s generalizability, we simultaneously monitor its performance in typical long CoT scenarios, such as General-Purpose Question-Answering (GPQA) [14] and Codeforces.
数据集。为了全面展示我们提出算法的有效性，我们在美国邀请赛数学考试（AIME）问题上进行实验，这些问题通常需要长时间的思考来解决。测试集包括最近两年的 AIME 问题。相反，训练集由所有过去的 AIME 竞赛问题组成，并补充了一些人工构造的困难数学问题。为了评估模型的可推广性，我们同时监控其在典型长 CoT 场景中的性能，例如通用问答（GPQA）[14]和 Codeforces。

Cold Start. This phase aims to enhance the model’s reasoning capabilities within a specific reasoning format. We used dozens of samples with a format that requires the model to place its thinking process between <think> and </think> tags before presenting the final answer. These samples were used to fine-tune the Qwen2.5 32B model [13], which we employ in our experiments for better reproducibility.
冷启动。此阶段旨在增强模型在特定推理格式下的推理能力。我们使用了数十个样本，这些样本要求模型在呈现最终答案之前，将其思考过程放置在和标签之间。这些样本用于微调 Qwen2.5 32B 模型[13]，我们在实验中使用该模型以提高可重复性。

Reward Modeling. We adopt the methodology commonly used in classical reasoning tasks across domains such as mathematics, code, and logical reasoning. This approach utilizes rule-based rewards to guide the learning process. When assigning the reward score, the verifier ignores the thinking part enclosed by the <think> tokens and extracts only the answer part for evaluation. Correct answers are assigned a score of $1.0$, while incorrect answers receive a score of $-1.0$.
奖励建模。我们采用在数学、代码和逻辑推理等领域的经典推理任务中常用的方法论。这种方法利用基于规则的奖励来指导学习过程。在分配奖励分数时，验证者忽略由标记包围的思考部分，仅提取答案部分进行评估。正确答案被分配分数 $1.0$ ，而错误答案则获得分数 $-1.0$ 。

RL Baseline. In our experiments, we use verl [18] as our experimental framework. The Proximal Policy Optimization (PPO) algorithm described in [12] serves as the baseline, with $\lambda$ set to 0.95 by default. The learning rates for the policy model and the value model are set to $1\times 10^{-6}$ and $2\times 10^{-6}$, respectively. The KL penalty coefficient is set to 0 because the rule-based reward cannot be hacked in the same way as that of a general reward model. We adopt different context length settings of 8k and 16k for different purposes: the 16k setting is used for comparison with state-of-the-art results, and the 8k setting is used for ablation studies.
RL 基线。在我们的实验中，我们使用 verl [18] 作为我们的实验框架。在[12]中描述的近端策略优化（PPO）算法作为基线，默认将 $\lambda$ 设置为 0.95。策略模型和价值模型的学习率分别设置为 $1\times 10^{-6}$ 和 $2\times 10^{-6}$ 。由于基于规则的奖励不能像通用奖励模型那样被破解，因此将 KL 惩罚系数设置为 0。我们根据不同目的采用不同的上下文长度设置，8k 和 16k：16k 设置用于与最先进的结果进行比较，8k 设置用于消融研究。

Value-Pretraining. We freeze the policy model and set the Generalized Advantage Estimation (GAE) $\lambda$ to 1.0 to obtain an unbiased return. The other hyperparameters are the same as those of the baseline Proximal Policy Optimization (PPO). By saving the value model at different steps of value pretraining, we can acquire multiple initial value checkpoints for RL training. We also conduct ablation studies on these checkpoints in our experiments.
值预训练。我们冻结策略模型并将广义优势估计（GAE） $\lambda$ 设置为 1.0 以获得无偏回报。其他超参数与基线近端策略优化（PPO）相同。通过在不同步骤保存价值预训练的价值模型，我们可以获得多个 RL 训练的初始价值检查点。我们还在实验中对这些检查点进行了消融研究。

Decoupled-GAE. Due to the value oscillation and reward signal decay described in Section 3.3, $\lambda_{\text{critic}}$ is set to 1.0. Meanwhile, $\lambda_{\text{actor}}$ used in the policy is maintained at 0.95 to enable a fair comparison with the baseline PPO. Subsequently, we assign values to $\lambda_{\text{actor}}$ ranging from 0.9 to 1.0 to investigate its impact on the convergence of the policy, while $\lambda_{\text{critic}}$ remains at 1.0.
解耦-GAE。由于第 3.3 节中描述的价值振荡和奖励信号衰减， $\lambda_{\text{critic}}$ 被设置为 1.0。同时，用于策略中的 $\lambda_{\text{actor}}$ 保持为 0.95，以便与基线 PPO 进行公平比较。随后，我们将 $\lambda_{\text{actor}}$ 的值从 0.9 到 1.0 进行赋值，以研究其对策略收敛的影响，而 $\lambda_{\text{critic}}$ 保持为 1.0。

We conduct RL training on the Qwen-32B-Base model using the proposed Value-Calibrated Proximal Policy Optimization (VC-PPO) algorithm. We then compare our model with the well-established Generalized Proximal Policy Optimization (GRPO) algorithm, which is employed in the DeepSeek-R1 model [4]. This experiment utilizes a 16k context length to attain the state-of-the-art performance.
我们在 Qwen-32B-Base 模型上使用提出的价值校准近端策略优化（VC-PPO）算法进行 RL 训练。然后，我们将我们的模型与在 DeepSeek-R1 模型[4]中使用的成熟通用的近端策略优化（GRPO）算法进行比较。该实验使用 16k 的上下文长度以达到最先进的性能。

The results are presented in Table 1. The proposed VC-PPO algorithm significantly outperforms GRPO under the same experimental setting. Our primary objective is to optimize the model’s performance on the American Invitational Mathematics Examination (AIME), which consists of Olympiad-level math problems. Consequently, the majority of the training data is math-related, and VC-PPO demonstrates the most substantial advantage on the AIME dataset.
结果展示在表 1 中。所提出的 VC-PPO 算法在相同的实验设置下显著优于 GRPO。我们的主要目标是优化模型在美国邀请赛数学考试（AIME）上的性能，该考试包含奥林匹克级别的数学问题。因此，大部分训练数据与数学相关，VC-PPO 在 AIME 数据集上展现出最显著的优势。

To the best of our knowledge, a pass@1 score of 48.8 on the AIME dataset stands as the highest performance attained by a Qwen-32B-Base model without employing distillation techniques. This score surpasses the AIME score of 47.0 reported in the DeepSeek-R1 technical report [4] under comparable experimental settings¹¹1It should be noted that a direct comparison between these two results is not entirely feasible because the dataset used for RL training in DeepSeek-R1 has not been made publicly available.. The increasing pass rate of the AIME dataset during the training process is illustrated in Figure 4. Additionally, we have deployed the VC-PPO algorithm in our internal model, which has achieved an AIME score of 74.
据我们所知，在 AIME 数据集上，Qwen-32B-Base 模型在不使用蒸馏技术的情况下，取得了 48.8 的 pass@1 分数，这是该模型所达到的最高性能。这个分数超过了在 DeepSeek-R1 技术报告中报道的、在可比实验设置下取得的 47.0 的 AIME 分数[4]。图 4 展示了 AIME 数据集在训练过程中的通过率不断提高。此外，我们还在内部模型中部署了 VC-PPO 算法，该算法实现了 74 的 AIME 分数。

For ablation studies, we use an 8k context length to enhance training efficiency.
对于消融研究，我们使用 8k 的上下文长度以提高训练效率。

In Table 2, we showcase the ablation results of Value-Pretraining and Decoupled-GAE. When directly applying the Proximal Policy Optimization (PPO) algorithm, it fails to improve the performance of the pre-trained model. This is because the output length of the model collapses. In contrast, the proposed Value-Calibrated Proximal Policy Optimization (VC-PPO) algorithm demonstrates a significant performance boost, highlighting its superiority in handling tasks that demand a long CoT.
在表 2 中，我们展示了 Value-Pretraining 和 Decoupled-GAE 的消融结果。当直接应用近端策略优化（PPO）算法时，它未能提高预训练模型的表现。这是因为模型输出的长度崩溃了。相比之下，提出的价值校准近端策略优化（VC-PPO）算法表现出显著的性能提升，突显了其在处理需要长 CoT 的任务中的优越性。

Moreover, when we conduct ablation experiments by removing either the Value-Pretraining or the Decoupled-GAE component from VC-PPO, there is a notable drop in performance. This decline emphasizes the crucial roles that both Value-Pretraining and Decoupled-GAE play in the effectiveness of our proposed VC-PPO algorithm.
此外，当我们通过移除 VC-PPO 中的 Value-Pretraining 或 Decoupled-GAE 组件进行消融实验时，性能明显下降。这种下降强调了 Value-Pretraining 和 Decoupled-GAE 在所提出的 VC-PPO 算法有效性中的关键作用。

In Figure 3, we compare the model performance at the same step in this ablation experiment, specifically after 100 steps of training. The decision to conduct this comparison is driven by the evident divergence in performance trends. The optimal configuration involves pretraining the value model for 100 steps. This is because additional training beyond this point might induce overfitting, which could negatively impact the model’s generalization ability.
图 3 中，我们比较了该消融实验中同一步骤下的模型性能，具体是在训练 100 步之后。进行这种比较的决定是由性能趋势的明显分歧所驱动的。最佳配置涉及在 100 步前对价值模型进行预训练。这是因为超过这个点进行额外的训练可能会引起过拟合，这可能会对模型的泛化能力产生负面影响。

The results of the ablation study on $\lambda_{\text{actor}}$ are presented in Table 4. It should be highlighted that all experimental groups with $\lambda_{\text{actor}}<1.0$ significantly outperform the group with $\lambda_{\text{actor}}=1.0$. This outcome supports our analysis in Section 3.3. In the case of the American Invitational Mathematics Examination (AIME), the experimental group with $\lambda_{\text{actor}}=0.99$ outperforms the other groups with lower $\lambda_{\text{actor}}$ values. However, there is only a slight decrease in performance within a certain range between $0.95$ and $1.0$. Therefore, the recommended setting for $\lambda_{\text{actor}}$ is $\lambda_{\text{actor}}\in[0.95,1.0)$.
消融研究的结果在表 4 中呈现。应强调，所有包含 $\lambda_{\text{actor}}<1.0$ 的实验组在性能上显著优于包含 $\lambda_{\text{actor}}=1.0$ 的组。这一结果支持了我们在第 3.3 节的分析。在美國邀請賽數學考試（AIME）的情况下，包含 $\lambda_{\text{actor}}=0.99$ 的实验组在性能上优于其他具有较低 $\lambda_{\text{actor}}$ 值的组。然而，在 $0.95$ 和 $1.0$ 之间的一定范围内，性能仅有轻微下降。因此，推荐的 $\lambda_{\text{actor}}$ 设置是 $\lambda_{\text{actor}}\in[0.95,1.0)$ 。

A smooth initial state for training is crucial in RLHF, especially in long CoT tasks.
训练 RLHF 中，一个平滑的初始状态至关重要，尤其是在长 CoT 任务中。

In traditional RL, both the value function and the policy are typically initialized randomly. However, in RLHF, the initial policy is usually initialized from the supervised fine-tuning (SFT) policy. This SFT policy acts as a strong prior for the learning process. In long CoT tasks, the initial policy is further enhanced with the CoT pattern, offering an even stronger prior.
在传统的强化学习中，值函数和策略通常随机初始化。然而，在 RLHF 中，初始策略通常从监督微调（SFT）策略初始化。这种 SFT 策略作为学习过程的一个强大先验。在长 CoT 任务中，初始策略通过 CoT 模式进一步增强，提供了一个更强大的先验。

Our empirical observations suggest that as the prior policy becomes stronger, it is increasingly essential to align the value model with the policy. Otherwise, the painstakingly constructed CoT pattern can easily be disrupted, as demonstrated in Figure 1. In our experiment, after applying the value-pretraining technique, which effectively aligns the value model with the initial policy, the collapse in output length is no longer observed. This result clearly highlights the significance of having a fully-aligned value model, as shown in Figure 5.
我们的实证观察表明，随着先验策略的增强，将价值模型与策略对齐变得越来越重要。否则，精心构建的 CoT 模式很容易被破坏，如图 1 所示。在我们的实验中，在应用了将价值模型与初始策略有效对齐的价值预训练技术后，输出长度的下降现象不再出现。这一结果清楚地突出了拥有完全对齐的价值模型的重要性，如图 5 所示。

Value-Pretraining injects knowledge into the value model, which is a superior form of value warm-up.
价值预训练将知识注入价值模型，这是一种更高级的价值预热形式。

We present the value loss during value-pretraining in Figure 6, where a two-stage convergence pattern can be observed. In the first stage, there is a rapid decline in value loss. We interpret this stage as range alignment, which shares similarities with the commonly used value warm-up technique in RL. However, in the second stage, the value loss decreases at a slower pace. We interpret this stage as knowledge injection. In this stage, the model starts to learn which tokens are more advantageous, a crucial aspect that has often been overlooked in previous research. As shown in Table 3, this stage has a substantial impact on the final performance of our model.
我们在图 6 中展示了值预训练过程中的值损失，其中可以观察到两阶段收敛模式。在第一阶段，值损失迅速下降。我们将这一阶段解释为范围对齐，这与 RL 中常用的值预热技术相似。然而，在第二阶段，值损失的下降速度较慢。我们将这一阶段解释为知识注入。在这一阶段，模型开始学习哪些标记更有利，这是先前研究中常被忽视的一个关键方面。如表 3 所示，这一阶段对我们的模型最终性能有重大影响。

Value optimization dynamics are more tolerant to variance, which leads to different variance-bias preferences in value and policy.
价值优化动态对变化更为容忍，这导致了价值和策略中对变化和偏差的不同偏好。

Based on the experimental results presented in Table 2 and the ablation results in Table 4, we can conclude that the value model favors a larger $\lambda$, resulting in higher variance but lower bias. In contrast, the policy model prefers lower variance which means a lower $\lambda$ than $1.0$. Notably, a relatively lower bias is still required at the same time since the extremely variance can hurt the performance anyway. This finding implicitly suggests that regression-style loss objectives, such as the mean squared error (MSE) loss used in value optimization, are less sensitive to variance. Conversely, policy-gradient-style objectives are more likely to be adversely affected by variance. This could serve as a promising avenue for further research in RL or RLHF.
基于表 2 中展示的实验结果和表 4 中的消融结果，我们可以得出结论，价值模型倾向于更大的 $\lambda$ ，导致更高的方差但更低的偏差。相比之下，策略模型更喜欢较低的方差，这意味着 $\lambda$ 比 $1.0$ 低。值得注意的是，同时仍然需要相对较低的偏差，因为极端的方差无论如何都会损害性能。这一发现隐含地表明，回归式损失目标，如用于价值优化的均方误差（MSE）损失，对方差不太敏感。相反，策略梯度式目标更容易受到方差的不利影响。这可以作为 RL 或 RLHF 进一步研究的有希望的途径。