Exploring the Limit of Outcome Reward
for Learning Mathematical Reasoning
探索学习数学推理的成果奖励极限

When adopting a large language model (LLM) for mathematic reasoning, the input to the LLM policy is a textual math problem that prompts the LLM to output a multi-step reasoning trajectory consisting of multiple tokens as actions. During RL training, common practices [6, 23] conduct sampling on the LLM to produce multiple reasoning trajectories, assign binary feedback (0/1 reward) based solely on the correctness of their final answer, and perform corresponding policy optimization using the sampled trajectories with reward.
在采用大型语言模型（LLM）进行数学推理时，LLM策略的输入是一个文本数学问题，该问题提示LLM输出由多个标记作为动作的多步推理轨迹。在强化学习训练过程中，常见的做法[6, 23]对LLM进行采样以产生多个推理轨迹，仅根据最终答案的正确性分配二元反馈（0/1 奖励），并使用带有奖励的采样轨迹进行相应的策略优化。

Policy Optimization. Consider a Markov Decision Process (MDP) defined by the tuple $(\mathcal{S},\mathcal{A},P,r,\gamma)$, where $\mathcal{S}$ is a finite state space (e.g., contextual steps in mathematical reasoning), $\mathcal{A}$ is the action space (i.e. the token space of LLMs), $P(s^{\prime}|s,a)$ specifies the state transition dynamics, $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function, and $\gamma\in[0,1)$ denotes the discount factor.
策略优化。考虑由元组 $(\mathcal{S},\mathcal{A},P,r,\gamma)$ 定义的马尔可夫决策过程（MDP），其中 $\mathcal{S}$ 是有限状态空间（例如数学推理中的上下文步骤）， $\mathcal{A}$ 是动作空间（即 LLMs 的标记空间）， $P(s^{\prime}|s,a)$ 指定状态转移动态， $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ 是奖励函数， $\gamma\in[0,1)$ 表示折扣因子。

In this section, we focus on KL-regularized policy optimization, which maximizes the expected cumulative returns while regularizing the policy $\pi_{\theta}(\cdot|s)$ toward a reference policy $\pi_{0}(\cdot|s)$. The objective function is formulated as:
在这一节中，我们关注 KL 正则化策略优化，该优化在最大化期望累积回报的同时，将策略 $\pi_{\theta}(\cdot|s)$ 正则化到参考策略 $\pi_{0}(\cdot|s)$ 。目标函数被定义为：

with the state-action value function $Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty}\gamma^{k}r(s_{t+k},a_{t+k})\mid s_{t}=s,a_{t}=a\right]$ under vanilla policy $\pi$. This objective admits a closed-form solution for optimal policy $\pi^{*}$:
在原始策略 $\pi$ 下，状态-动作值函数 $Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty}\gamma^{k}r(s_{t+k},a_{t+k})\mid s_{t}=s,a_{t}=a\right]$ 的此目标函数对于最优策略 $\pi^{*}$ 具有闭式解。

where $Z(s)=\mathbb{E}_{a\sim\pi_{0}(\cdot|s)}\left[\exp\left(Q^{\pi}(s,a)/\alpha\right)\right]$ is the partition function that ensures normalization.
$Z(s)=\mathbb{E}_{a\sim\pi_{0}(\cdot|s)}\left[\exp\left(Q^{\pi}(s,a)/\alpha\right)\right]$ 是确保归一化的配分函数。

Best-of-N (BoN) Sampling. As a common and efficient strategy to sample multiple reasoning trajectories from LLMs, Best-of-$N$ sampling selects the trajectory with maximal reward among $n$ independent rollouts from $\pi_{0}$ to enhance policy performance. Formally, given candidate actions $\{a^{(i)}\}_{i=1}^{n}\sim\pi_{0}(\cdot|s)$, the chosen action is $a^{*}=\arg\max_{a^{(i)}}Q(s,a^{(i)})$. This strategy effectively leverages the exploration-exploitation trade-off through parallel sampling [24, 25].
Best-of-N (BoN) 样本选择。作为一种常见且高效的策略，从LLMs中采样多个推理轨迹，Best-of- $N$ 样本选择从 $\pi_{0}$ 到 $n$ 独立运行中选择具有最大奖励的轨迹以增强策略性能。形式上，给定候选动作 $\{a^{(i)}\}_{i=1}^{n}\sim\pi_{0}(\cdot|s)$ ，选择动作 $a^{*}=\arg\max_{a^{(i)}}Q(s,a^{(i)})$ 。该策略通过并行采样[24, 25]有效地利用了探索-利用权衡。

Binary Feedback under Outcome Supervision. Though a reasoning trajectory usually contains multiple reasoning steps with thousands of tokens, there lacks an efficient approach to automatically label the correctness of each token or reasoning step in math reasoning tasks. Thus, a practical way is to parse the final answer from the reasoning trajectory [13, 26], evaluate its correctness based on rules or models, and then provide an outcome reward at the end of the trajectory as below:
二进制反馈在结果监督下。尽管推理轨迹通常包含多个推理步骤，涉及数千个标记，但在数学推理任务中缺乏一种有效的方法来自动标注每个标记或推理步骤的正确性。因此，一种实用的方法是解析推理轨迹中的最终答案[13, 26]，根据规则或模型评估其正确性，然后在轨迹结束时提供结果奖励如下：

which intrinsically treats the correct trajectory equally for learning. Moreover, the reward signal is severely sparse when compared to the thousands of tokens and does not provide any signal of progress or correctness for intermediate steps. The resulting reward distribution of trajectories is also different from that of the dense reward function constructed through preference pairs in traditional RL for large language models [27], which induces a more appropriate optimization framework for mathematical reasoning tasks, discussed in the next section.
该算法内在地平等对待正确的轨迹以进行学习。此外，与数千个标记相比，奖励信号非常稀疏，并且不提供任何关于中间步骤的进展或正确性的信号。轨迹的奖励分布也与通过偏好对构建的传统强化学习中的密集奖励函数不同[27]，这为数学推理任务提供了一个更合适的优化框架，将在下一节中讨论。

Building upon the reward equivalence principle stated in Eq. 3, we first formalize a key probabilistic characteristic of BoN sampling:
基于式（3）中提出的奖励等价原理，我们首先形式化 BoN 采样的一种关键概率特性：

The proof follows directly from the union law of BoN sampling ($\text{BoN}_{n+m}=\text{BoN}_{2}(\text{BoN}_{m},\text{BoN}_{n})$) and the trivial distinguishability of $0-1$ rewards. This result reveals that for problems with attainable positive responses, we are using a BoN generator with an arbitrary sampling budget to construct the positive training samples.
证明直接来自 BoN 采样的并集定律（ $\text{BoN}_{n+m}=\text{BoN}_{2}(\text{BoN}_{m},\text{BoN}_{n})$ ）和 $0-1$ 奖励的平凡可区分性。这一结果揭示了对于具有可实现积极响应的问题，我们使用具有任意采样预算的 BoN 生成器来构建积极训练样本。

To quantify the distributional divergence induced by BoN sampling, prior work [28, 29, 30] has analyzed the KL divergence between the BoN distribution $\pi_{\text{BoN}}$ and the original policy $\pi$. For continuous trajectory spaces $\mathcal{S}$, the BoN distribution admits the explicit form:
为了量化 BoN 采样引起的分布差异，先前的研究[28, 29, 30]分析了 BoN 分布 $\pi_{\text{BoN}}$ 与原始策略 $\pi$ 之间的 KL 散度。对于连续轨迹空间 $\mathcal{S}$ ，BoN 分布可以表示为显式形式：

where $P(s)$ denotes the cumulative distribution function (CDF) associated with $\pi(s)$. The corresponding KL divergence is given by
$P(s)$ 表示与 $\pi(s)$ 相关的累积分布函数（CDF）。相应的 KL 散度由下式给出：

The KL divergence expression possesses two crucial properties, as $n\to\infty$, the $\text{KL}(\pi_{\text{BoN}}\parallel\pi)$ is strictly increasing in $n$ for $n\geq 1$, and converge to $\log n\to\infty$, covering the entire positive real axis. This implies that for any prescribed KL divergence constraint $\epsilon>0$, there exists a BoN parameterization for approximating the optimal policy, and can be simply sampled with the minimal $n(\epsilon)$ satisfying that
KL 散度表达式具有两个关键性质，即 $n\to\infty$ ， $\text{KL}(\pi_{\text{BoN}}\parallel\pi)$ 在 $n$ 上严格递增，对于 $n\geq 1$ 收敛到 $\log n\to\infty$ ，覆盖整个正实数轴。这意味着对于任何给定的 KL 散度约束 $\epsilon>0$ ，存在一个 BoN 参数化来逼近最优策略，并且可以通过满足最小 $n(\epsilon)$ 的简单采样来实现。

BoNBoN [31] empirically shows that BoN sampling achieves the optimal win rate under fixed KL constraint by exhaustive search over the positive support. Therefore, the behavior cloning on BoN-selected positive samples directly learns the analytic solution to Eq. 1. Intuitively, since every correct answer is preferred identically in the outcome-supervised sense, we only need to sample until we get a positive example, whose generating probability distribution will be the same as randomly picking from arbitrarily large numbers of samples.
BoNBoN [31] 通过对正支持集进行穷举搜索，在固定的 KL 约束下，实证表明 BoN 采样实现了最优的胜率。因此，在 BoN 选定的正样本上进行的行为克隆直接学习方程 1 的解析解。直观上，由于在结果监督的意义下，每个正确答案都是相同的，我们只需要采样直到得到一个正例，其生成概率分布将与从任意大量样本中随机选择相同。

Based on the theoretical understanding established, we formulate the first component of the learning objective in OREAL by incorporating KL-constrained max-likelihood-objective over positive examples obtained through sampling:
基于建立的理论理解，我们通过将 KL 约束下的最大似然目标函数应用于通过采样获得的正例，形成了 OREAL 学习目标的第一部分：

where $\mathcal{D}^{+}$ denotes the set of positive trajectories selected via BoN sampling from RL rollouts.
$\mathcal{D}^{+}$ 表示通过 BoN 采样从 RL 回放中选择出的正轨迹集合。

As established in Section 2.2, direct behavioral cloning on positive responses can effectively recover the policy distribution. BOND [32] proposes estimating Jeffreys divergence [33] for the BoN strategy to train with both positive and negative samples, and demonstrates that signals from unsuccessful trajectories provide critical information about decision boundaries and failure modes.
如第 2.2 节所述，直接在积极反应上进行行为克隆可以有效地恢复策略分布。BOND [32] 提出为 BoN 策略估计 Jeffreys 距离 [33]，以训练正负样本，并证明不成功轨迹的信号提供了关于决策边界和故障模式的关键信息。

In this section, we will discuss the relationship between the BoN (Best-of-N) distribution and the optimization objective defined in Eq. 1, then elucidate the necessity of reward reshaping when training with negative samples. Notably, while Eq. 4 shares structural similarities with Eq. 2, its application to mathematical reasoning tasks with binary feedback requires reformulation. Specifically, the transformed BoN distribution can be expressed as
在本节中，我们将讨论 BoN（N 中最佳）分布与式（1）中定义的优化目标之间的关系，然后阐述在训练负样本时调整奖励的必要性。值得注意的是，虽然式（4）与式（2）在结构上相似，但其应用于具有二元反馈的数学推理任务时需要重新表述。具体来说，转换后的 BoN 分布可以表示为：

which reveals fundamental differences between the BoN distribution and the original sampling distribution. Consider a scenario where two correct and two incorrect solutions are sampled, yielding an empirical accuracy of 50%. However, the probability of selecting negative samples under Best-of-4 becomes $(0.5)^{4}=6.25\%$, significantly lower than the original distribution. This discrepancy necessitates reward shaping to maintain consistency between our optimization target and the expected return under the BoN distribution.
揭示 BoN 分布与原始采样分布之间的基本差异。考虑一种情况，其中采样了两个正确和两个错误的解决方案，得到 50%的经验准确性。然而，在 Best-of-4 下选择负样本的概率为 $(0.5)^{4}=6.25\%$ ，显著低于原始分布。这种差异需要调整奖励形状，以保持我们的优化目标与 BoN 分布下的预期回报之间的一致性。

Building on BoN-RLB’s [34] application of the log-likelihood trick for BoN-aware policy gradients, we analyze the reward shaping technic for negative samples to maintain gradient consistency with Section 2.2. With expectation return $p$ follow the definition in Lemma 2.1. The policy gradient under the BoN distribution can be derived as
基于 BoN-RLB[34]对 BoN 感知策略梯度的对数似然技巧的应用，我们分析了用于负样本的奖励塑造技术，以保持与第 2.2 节的梯度一致性。期望回报 $p$ 遵循引理 2.1 的定义。在 BoN 分布下的策略梯度可以推导如下：

where $I_{D_{+}}(s)$ and $I_{D_{-}}(s)$ denote indicator functions for positive and negative sample sets respectively. Notably, these indicators are independent of policy parameters $\theta$. Given $\mathbb{E}_{s\sim\pi_{\text{bon}}}[I_{D_{+}}(s)]=1-(1-p)^{n}$, we derive the gradient components as
$I_{D_{+}}(s)$ 和 $I_{D_{-}}(s)$ 分别表示正样本集和负样本集的指示函数。值得注意的是，这些指示函数与策略参数 $\theta$ 无关。给定 $\mathbb{E}_{s\sim\pi_{\text{bon}}}[I_{D_{+}}(s)]=1-(1-p)^{n}$ ，我们推导出梯度分量如下：

Similarly, we also have  同样，我们也有

This derivation reveals that when assigning unit reward ($R(s)=1$) to positive samples, gradient consistency requires reshaping negative sample rewards to $R^{\star}(s)\triangleq(1-p)R(s)$. Based on this reward shaping, we can construct policy optimization both on positive and negative samples for optimal policy.
这一推导表明，在将单位奖励（ $R(s)=1$ ）分配给正样本时，梯度一致性要求将负样本奖励重塑为 $R^{\star}(s)\triangleq(1-p)R(s)$ 。基于这种奖励重塑，我们可以构建针对正负样本的政策优化，以获得最佳策略。

To obtain the parameter $1-p$ which can be linked to the Monte-Carlo (MC) advantage estimation, we can simply estimate that probability by calculating the expected accuracy on the sample space by counting a small number of responses. In this paper we apply a similar setting to RLOO [35], namely $R_{RLOO}(s)=R(s)-\frac{1}{N-1}\sum_{s^{\star}\neq s}R(s^{\star})$ for unbiased mean reward and train with policy gradient. The second part of our OREAL objective is then formulated as below:
为了获得可以与蒙特卡洛（MC）优势估计相联系的参数 $1-p$ ，我们可以简单地通过计算样本空间上的期望精度并计数少量响应来估计该概率。在本文中，我们应用了与 RLOO [35]类似的设置，即 $R_{RLOO}(s)=R(s)-\frac{1}{N-1}\sum_{s^{\star}\neq s}R(s^{\star})$ 用于无偏平均奖励，并使用策略梯度进行训练。我们的 OREAL 目标函数的第二部分如下：

where $p=\mathbb{P}_{\theta\sim\pi}[R(\theta)=1]$, $S_{-}$ is the failed subset generated by policy model, and $F$ represents the preprocessing for advantage scores to serve as a generalized form, for example, $F(1-p)\triangleq\frac{r_{i}-mean(\{r_{i}...r_{n}\})}{std(\{r_{i}...r_{n}\})}$ in the recent GRPO [6] algorithm, where $mean(\{r_{i}...r_{n}\})\to p$ when $n\to\infty$.
$p=\mathbb{P}_{\theta\sim\pi}[R(\theta)=1]$ 、 $S_{-}$ 表示由策略模型生成的失败子集， $F$ 代表预处理优势分数，以作为通用形式，例如，在最近的 GRPO [6] 算法中的 $F(1-p)\triangleq\frac{r_{i}-mean(\{r_{i}...r_{n}\})}{std(\{r_{i}...r_{n}\})}$ ，其中当 $mean(\{r_{i}...r_{n}\})\to p$ 时， $n\to\infty$ 。

In the previous discussion, we introduced the adaptation of binary reward training in response space. However, since the outcome supervision only provides feedback at the sequence level, this modeling essentially reduces to a contextual bandit without internal reward modeling within the MDP. A common counterexample is PPO, which utilizes a separate critic model to estimate the value function. However, such a solution appears to be expensive and complex, which has induced numerous explorations on how to stabilize the PPO training.
在前一次讨论中，我们介绍了在响应空间中对二进制奖励训练的适应。然而，由于结果监督仅在序列级别提供反馈，这种建模本质上降低为 MDP 内部没有内部奖励模型的上下文多臂老虎机。一个常见的反例是 PPO，它使用一个单独的评论家模型来估计价值函数。然而，这种解决方案似乎既昂贵又复杂，这引发了众多关于如何稳定 PPO 训练的探索。

Things become slightly different in mathematical reasoning, where the model can spontaneously revise omissions in intermediate steps to obtain the correct final answer. Therefore, outcome supervision is preferred, and the value function is more meant to be a simple credit assignment to determine how much the process step contributes to the outcome reward. With efficiency and performance trade-off considerations, we choose to use some low-cost alternatives for sequence-level reweighting.
事物在数学推理中略有不同，模型可以自发地修正中间步骤中的遗漏以获得正确的最终答案。因此，结果监督更受欢迎，价值函数更多的是一种简单的信用分配，以确定过程步骤对结果奖励的贡献程度。考虑到效率和性能权衡，我们选择使用一些低成本的替代方案进行序列级重新加权。

Taking into account the deterministic dynamics in mathematical reasoning ($s_{t+1}=f(s_{t},a_{t})$), the state-action function $Q^{\pi}(s_{<t},\pi(s_{t}))$ simplifies to the cumulative discounted reward of policy $\pi$:
考虑到数学推理中的确定性动力学（ $s_{t+1}=f(s_{t},a_{t})$ ），状态-动作函数 $Q^{\pi}(s_{<t},\pi(s_{t}))$ 简化为策略 $\pi$ 的累积折现奖励：

Since intermediate rewards are not provided in mathematical reasoning tasks, we define an advantage function based solely on outcome feedback:
由于数学推理任务中不提供中间奖励，我们定义了一个仅基于结果反馈的优势函数：

This formulation treats $A(s_{\leq t})$ as a token-wise credit assignment mechanism, estimating each token’s contribution toward the final outcome.
这种公式将 $A(s_{\leq t})$ 视为一种按词元分配信用的机制，估计每个词元对最终结果贡献的大小。

For a pair of responses $y_{1}$ and $y_{2}$ to the same query, their initial values coincide $V_{0}^{1}=V_{0}^{2}$. The win rate between them then satisfies:
对于同一查询的响应对 $y_{1}$ 和 $y_{2}$ ，它们的初始值相同 $V_{0}^{1}=V_{0}^{2}$ 。它们之间的胜率满足：

Equation 9 indicates that for any function family $\mathcal{A}=\{A(s_{\leq t})\}$ , a cumulative reward function through sequence aggregation can be constructed to model rewards:
方程 9 表明，对于任何函数族 $\mathcal{A}=\{A(s_{\leq t})\}$ ，可以通过序列聚合构建一个累积奖励函数来模拟奖励：

which is trainable via preference pairs $\{(y_{w},y_{l})\}$ by fitting the outcome feedback. The learned $A(s_{\leq t})$ serves as a weighting function for credit assignment, which is used to reweight the original training loss, emphasizing critical reasoning steps or errors. An analogous implementations is $r2Q^{*}$ [36, 37] by defining $A=\log\frac{\pi(y_{i})}{\pi_{\text{ref}}(y_{i})}$, PRIME [20] then apply this formulation to improve performance of RLOO. In our work, following the practice from [38], we directly train a token-level reward function $w(s_{\leq t})$ satisfying
通过偏好对 $\{(y_{w},y_{l})\}$ 进行训练，通过拟合结果反馈。学习到的 $A(s_{\leq t})$ 作为信用分配的加权函数，用于重新加权原始训练损失，强调关键推理步骤或错误。类似的实现是 $r2Q^{*}$ [ 36, 37]，通过定义 $A=\log\frac{\pi(y_{i})}{\pi_{\text{ref}}(y_{i})}$ ，PRIME [ 20]，然后将此公式应用于提高 RLOO 的性能。在我们的工作中，遵循 [ 38] 的做法，我们直接训练一个满足 $w(s_{\leq t})$ 的标记级奖励函数。

without constraining KL-divergence to reference model in reward model training. These sequential rewards can serve as a proxy for the contribution of thinking steps to the result accuracy. Assuming a pair of prefix-consistent correct and incorrect samples, due to the causal inference nature of the token-level reward model, the preference optimization for these samples will only function on the steps that have different contents, which induces higher credits on the core reasoning step that affects the final result. We further discuss the training details of this model and analyze the visualization of its token-wise scoring effects later in Section 3.2 and Appendix A.
不将 KL 散度约束在奖励模型训练中的参考模型上。这些序列奖励可以作为思维步骤对结果准确度贡献的代理。假设有一对前缀一致的正确和错误样本，由于标记级奖励模型的因果推断性质，对这些样本的偏好优化将仅作用于内容不同的步骤，这导致对影响最终结果的核心推理步骤赋予更高的信用。我们将在第 3.2 节和附录 A 中进一步讨论该模型的训练细节，并分析其标记级评分效果的可视化。

In practice, we decompose the output weight $w(s)$ for positive and negative samples and clip on the positive axis to prevent reversing the direction of the optimized gradient, denoted as $\omega^{+}$ and $\omega^{-}$:
在实践中，我们将正负样本的输出权重 $w(s)$ 进行分解，并在正轴上截断以防止优化梯度的方向反转，分别表示为 $\omega^{+}$ 和 $\omega^{-}$ ：

Giving input query $d$, the overall loss is as follows:
输入查询 $d$ ，整体损失如下：

where $\eta$ represents the balancing weights for positive and negative losses.
$\eta$ 代表正负损失的平衡权重。

We utilize Qwen2.5-7B and Qwen2.5-32B [39] as the base model. Initially, we fine-tune the base models using long chain-of-thought data obtained through rejection sampling [23]. This rejection sampling fine-tuned (RFT) [23] models then serve as the initialization for the policy model in our RL framework. We also explore to use of DeepSeek-R1-Distill-Qwen-7B [13] as the initial policy model and perform OREAL on it and discuss the influence of different initial policy models in Section 4.4. The training data for the RFT models consists of in-house datasets supported by OpenDataLab [40] and open-source datasets including Numina [41] and the training set of MATH [21].
我们使用 Qwen2.5-7B 和 Qwen2.5-32B[39]作为基础模型。最初，我们使用通过拒绝采样[23]获得的长链思维数据对基础模型进行微调。这种拒绝采样微调（RFT）[23]模型随后作为我们强化学习框架中策略模型的初始化。我们还探索使用 DeepSeek-R1-Distill-Qwen-7B[13]作为初始策略模型，并对其实施 OREAL，并在第 4.4 节中讨论不同初始策略模型的影响。RFT 模型的训练数据包括由 OpenDataLab[40]支持的内部数据集和包括 Numina[41]和 MATH 训练集的开源数据集。

where $s$ represents the sampled trajectory, $r\in\{0,1\}$ is the binary outcome reward from the verifier, and $p(s)=\sigma(\frac{1}{T}\sum_{t}^{T}w(s_{t}))$ denotes the predicted probability of correctness by the token-level reward model $w$.
$s$ 表示采样轨迹， $r\in\{0,1\}$ 是验证器的二进制结果奖励， $p(s)=\sigma(\frac{1}{T}\sum_{t}^{T}w(s_{t}))$ 表示由标记级奖励模型 $w$ 预测的正确性概率。

To further analyze the behavior of the token-level reward model, we visualize its output distribution $w(s_{t})$ during the on-policy RL training process (see Appendix A). In this training paradigm, $w(s_{t})$ assigns token-wise importance scores across the chain-of-thought reasoning process, capturing each token’s contribution to the final correctness of the generated response. Consequently, this allows us to leverage $w(s_{t})$ for importance sampling during the optimization process, enabling a more principled selection of informative tokens.
为了进一步分析基于标记级别的奖励模型的行为，我们在在线策略强化学习训练过程中可视化其输出分布（见附录 A）。在这种训练范式下， $w(s_{t})$ 在整个思维链推理过程中分配标记的重要性分数，捕捉每个标记对生成响应最终正确性的贡献。因此，这使我们能够在优化过程中利用 $w(s_{t})$ 进行重要性采样，从而实现更有原则的信息标记选择。

During training iterations, each batch consists of 64 questions, with 16 rollouts per question. The max length of each rollout trajectory is set to 16384 tokens. Then the correctness of each response is averaged to calculate the pass rate, and questions with an overall pass rate of 0 or 1 are discarded. For the remaining trajectories, we retain only one correct response and one incorrect response per question, ensuring a balanced distribution of positive and negative samples for token-level reward model training.
在训练迭代过程中，每个批次包含 64 个问题，每个问题进行 16 次 rollout。每个 rollout 轨迹的最大长度设置为 16384 个 token。然后，将每个响应的正确性平均计算通过率，对于整体通过率为 0 或 1 的问题予以丢弃。对于剩余的轨迹，每个问题仅保留一个正确响应和一个错误响应，以确保正负样本在 token 级奖励模型训练中的平衡分布。

For optimization, the policy model is trained with a learning rate of $5e{-7}$, while the token-level reward model is trained with a learning rate of $2e{-6}$. The latter undergoes a 10-step warm-up phase before training begins. Both models employ a cosine annealing learning rate schedule, decaying to $1/5$ of the initial learning rate over time. We optimize both models using the AdamW optimizer. The total number of training steps is 80, with evaluation conducted every 10 steps. The KL coefficient $\beta$ is set to 0.01. We select the best-performing model determined by evaluation metrics.
为了优化，策略模型以学习率 $5e{-7}$ 进行训练，而令牌级奖励模型以学习率 $2e{-6}$ 进行训练。后者在训练开始前经过 10 步预热阶段。两个模型都采用余弦退火学习率计划，学习率随时间衰减至初始学习率的 $1/5$ 。我们使用 AdamW 优化器优化这两个模型。总训练步数为 80 步，每 10 步进行一次评估。KL 系数 $\beta$ 设置为 0.01。我们选择根据评估指标确定的最佳性能模型。

During the RL training procedure, we observe that the model consistently struggles with certain types of questions, particularly those involving specific knowledge and skill areas, such as trigonometric constant transformations, probability statistics, series transformations, etc. We believe this is caused by the insufficient learning of the base model on these concepts in the Pre-training or RFT stages.
在 RL 训练过程中，我们观察到模型在处理某些类型的问题时一直存在困难，尤其是涉及特定知识和技能领域的问题，如三角恒等变换、概率统计、级数变换等。我们认为这是由于基础模型在预训练或 RFT 阶段对这些概念学习不足所致。

To address this problem, we implement a skill-based enhancement approach, using the MATH dataset to reduce the high cost of skill annotation. Specifically, we annotate each question in the training set with its corresponding core skill. For questions that the model repeatedly fails to answer correctly during the RL phase, we perform data augmentation by including similar questions from the training set that share the same skill. These augmented questions are then added to the training data during the RFT stage to help the model better internalize these skills.
为了解决这个问题，我们实施了一种基于技能的增强方法，利用 MATH 数据集来降低技能标注的高成本。具体来说，我们将训练集中的每个问题标注与其对应的核心技能。对于在 RL 阶段模型反复无法正确回答的问题，我们通过包括来自训练集中具有相同技能的相似问题来进行数据增强。然后，将这些增强问题添加到 RFT 阶段的训练数据中，以帮助模型更好地内化这些技能。

Tabel 1 shows the results of the comprehensive evaluation, highlighting the performance of our proposed models across different parameter scales. Notably, at the 7B scale, OREAL-7B achieves a remarkable pass@1 accuracy of 91.0 on the MATH-500 and 59.9 on OlympiadBench. To the best of our knowledge, this is the first time a model of this size has reached such a high level of accuracy using RL instead of distillation. This performance not only establishes a new milestone for RL-based methods but also surpasses significantly larger models, including QwQ-32B-Preview and OpenAI-o1-mini, demonstrating the effectiveness of our approach. Furthermore, after applying OREAL on the previous best 7B model, DeepSeek-R1-Distill-Qwen-7B, the resulting model, OREAL-DSR1-Distill-Qwen-7B, obtains 94.0 and 66.1 pass@1 accuracy on MATH-500 and OlympiadBench, respectively, setting new records among all 7B models. This result verifies the effectiveness of OREAL even when faced with strong initial policies.
表 1 展示了全面评估的结果，突出了我们提出的模型在不同参数尺度下的性能。值得注意的是，在 7B 尺度下，OREAL-7B 在 MATH-500 上实现了令人瞩目的 91.0 pass@1 准确率，在 OlympiadBench 上为 59.9。据我们所知，这是首次有如此规模的模型使用强化学习而非蒸馏达到如此高的准确率。这一性能不仅为基于强化学习的方法树立了新的里程碑，而且显著超越了包括 QwQ-32B-Preview 和 OpenAI-o1-mini 在内的更大规模模型，证明了我们方法的有效性。此外，在将 OREAL 应用于之前最佳的 7B 模型 DeepSeek-R1-Distill-Qwen-7B 之后，得到的模型 OREAL-DSR1-Distill-Qwen-7B 在 MATH-500 和 OlympiadBench 上分别获得了 94.0 和 66.1 的 pass@1 准确率，在所有 7B 模型中创造了新纪录。这一结果验证了 OREAL 即使在面对强大的初始策略时也具有有效性。

For 32B models, OREAL-32B achieves a groundbreaking pass@1 accuracy of 95.0 on MATH-500, 46.7 on AIME2025-I, 74.8 on LiveMathBench, and 72.4 on OlympiadBench, setting a new state-of-the-art among all previously reported models. These results underscore the advantages of our methodology, including its scalability for training superior mathematical reasoning models across different model sizes.
对于 32B 模型，OREAL-32B 在 MATH-500 上实现了突破性的 95.0% pass@1 准确率，在 AIME2025-I 上为 46.7%，在 LiveMathBench 上为 74.8%，在 OlympiadBench 上为 72.4%，在所有已报道的模型中创造了新的最高水平。这些结果突显了我们方法的优势，包括其训练不同规模数学推理模型的可扩展性。

Compared to the most competitive baseline, DeepSeek-R1-Distill-Qwen series, OREAL-32B demonstrates a clear advantage, whereas OREAL-7B lags slightly behind than the distilled 7B model, despite being trained on the same dataset as OREAL-32B. We attribute this discrepancy to the different affinities of the base models for the post-training data. Qwen-7B and Qwen-32B may exhibit varying degrees of knowledge gaps due to model sizes and pre-training settings. Our training data appears to better complement the existing knowledge of Qwen-32B, while it may be less effective in bridging gaps for Qwen-7B.
与最具竞争力的基线模型 DeepSeek-R1-Distill-Qwen 系列相比，OREAL-32B 展现出明显的优势，而 OREAL-7B 尽管与 OREAL-32B 在相同的数据集上训练，但在蒸馏后的 7B 模型上略逊一筹。我们将这种差异归因于基模型对训练后数据的亲和力不同。由于模型大小和预训练设置的不同，Qwen-7B 和 Qwen-32B 可能表现出不同程度的知识差距。我们的训练数据似乎更好地补充了 Qwen-32B 的现有知识，而在弥合 Qwen-7B 的知识差距方面可能效果较差。

In addition, OREAL-DSR1-Distill-Qwen-7B improves the MATH-500 score from 92.8 to 94.0 and also achieves gains on LiveMathBench and OlympiadBench. However, its performance on the AIME benchmark series is comparatively weaker. We observe the same disadvantages of OREAL-32B and OREAL-7B, whose AIME2024 scores are relatively lower than the best scores. Since the overall performance verifies the effectiveness of the OREAL algorithm, we attribute the reason to the deficiency (e.g., in response quality, query difficulty, and quantity) of RFT data and RL training queries for obtaining high performance in the domain of AIME and leave it for the future work.
此外，OREAL-DSR1-Distill-Qwen-7B 将 MATH-500 分数从 92.8 提高到 94.0，并在 LiveMathBench 和 OlympiadBench 上取得进步。然而，其在 AIME 基准系列上的性能相对较弱。我们观察到 OREAL-32B 和 OREAL-7B 相同的缺点，其 AIME2024 分数相对低于最佳分数。由于整体性能验证了 OREAL 算法的有效性，我们将原因归因于 RFT 数据（例如，在响应质量、查询难度和数量方面）和 RL 训练查询的不足，以获得 AIME 领域的较高性能，并将其留待未来工作。

To verify the effectiveness of each component described in Section 2, we progressively add the proposed component based on the 7B model and compare the evaluation results on MATH-500, starting from REINFORCE [50] as baseline.
为了验证第 2 节中描述的每个组件的有效性，我们逐步添加基于 7B 模型的所提组件，并从 REINFORCE [50]作为基线开始，在 MATH-500 上比较评估结果。

As shown in Tabel 2, we add each component step by step, where “Reward Shaping” represents $L_{2}$ introduced in Section 2.3, “Behavior Cloning” represents $L_{1}$ introduced in Section 2.2, “Importance Shaping” represents $L_{total}$ introduced in Section 2.4. The gradual addition of the modules steadily increases the Pass@1 scores of the 7B model on MATH-500, proving the effectiveness of our method. Ultimately, the policy model is raised from an initial score of 84.8 to 91.0.
如表 2 所示，我们逐步添加每个组件，其中“奖励塑造”代表第 2.3 节中引入的 $L_{2}$ ，“行为克隆”代表第 2.2 节中引入的 $L_{1}$ ，“重要性塑造”代表第 2.4 节中引入的 $L_{total}$ 。模块的逐步添加稳步提高了 7B 模型在 MATH-500 上的 Pass@1 分数，证明了我们方法的有效性。最终，策略模型从初始分数 84.8 提升到 91.0。

We also report average pass@1 accuracy across all benchmarks during the training process with different RL settings. As shown in Figure 2, the REINFORCE training process is unstable, which can be mitigated by “Reward Shaping”. “Behavioral Cloning” for positive samples can speed up convergence and show better performance early in training. Although the performance growth of “Importance Sampling” is relatively slow in the early stage of training, it ultimately obtains the best results.
我们还在训练过程中报告了不同强化学习设置下所有基准的平均 pass@1 准确率。如图 2 所示，REINFORCE 训练过程不稳定，可以通过“奖励塑造”来缓解。对于正样本的“行为克隆”可以加快收敛速度，并在训练早期表现出更好的性能。尽管“重要性采样”在训练早期性能增长相对较慢，但最终获得最佳结果。

We further analyze OREAL by adopting it to several different initial policy models, as shown in Table 3. OREAL consistently improves the performance of each initial policy model, including our own trained model and the strong distilled model [13], on MATH-500, LiveMathBench, and OlympiadBench, except slight fluctuations on AIME2024 and AIME2025 part1 when the performance of initial policy models are already high (e.g., DeepSeek-R1-Distill-Qwen-7B), which demonstrates the generality of OREAL.
我们进一步通过将 OREAL 应用于多个不同的初始策略模型来分析它，如表 3 所示。OREAL 在 MATH-500、LiveMathBench 和 OlympiadBench 上始终提高了每个初始策略模型的表现，包括我们自己的训练模型和强大的蒸馏模型[13]，除了在初始策略模型性能已经很高时（例如，DeepSeek-R1-Distill-Qwen-7B）在 AIME2024 和 AIME2025 部分 1 上略有波动，这证明了 OREAL 的通用性。

After adding skill-based enhancement data (introduced in Section 3.3), there is a significant rise in MATH-500 scores for the initial policy model (row 1 and row 3) and the corresponding RL-trained model (row 2 and row 4). Since our enhancement is performed primarily for the MATH-500, this verifies the effectiveness of the skill-based enhancement approach. In addition, the performance of the model after RL is strongly correlated with the capabilities of the initial policy model itself. The stronger the initial policy model, the higher the performance that RL can deliver, indicating the importance of policy initialization.
在添加基于技能的增强数据（如第 3.3 节所述）后，初始策略模型（第 1 行和第 3 行）以及相应的 RL 训练模型（第 2 行和第 4 行）的 MATH-500 分数显著上升。由于我们的增强主要针对 MATH-500，这验证了基于技能的增强方法的有效性。此外，经过 RL 训练后的模型性能与初始策略模型本身的能力密切相关。初始策略模型越强，RL 能带来的性能越高，这表明策略初始化的重要性。