s1: Simple test-time scaling
s1: 简单测试时间缩放

Our largest source is NuminaMATH (LI et al., 2024) with 30,660 mathematical problems from online websites. We also include historical AIME problems (1983-2021). To enhance diversity, we add OlympicArena (Huang et al., 2024a) with 4,250 questions spanning Astronomy, Biology, Chemistry, Computer Science, Geography, Mathematics, and Physics from various Olympiads. OmniMath (Gao et al., 2024a) adds 4,238 competition-level mathematics problems. We also include 2,385 problems from AGIEval (Zhong et al., 2023), which features questions from standardized tests like SAT and LSAT, covering English, Law, and Logic. We refer to Table 7 in §C for our other sources.
我们的最大来源是 NuminaMATH（LI 等，2024），包含来自在线网站的 30,660 个数学问题。我们还包括 1983-2021 年的历史 AIME 问题。为了增加多样性，我们添加了 OlympicArena（Huang 等，2024a），包含来自各种奥林匹克竞赛的天文学、生物学、化学、计算机科学、地理学、数学和物理学等领域的 4,250 个问题。OmniMath（Gao 等，2024a）增加了 4,238 个竞赛级别的数学问题。我们还包括来自 AGIEval（Zhong 等，2023）的 2,385 个问题，这些问题是来自 SAT 和 LSAT 等标准化考试的问题，涵盖英语、法律和逻辑。我们参考§C 中的表 7 以了解其他来源。

To complement these existing datasets, we create two original datasets. s1-prob consists of 182 questions from the probability section of Stanford University’s Statistics Department’s PhD Qualifying Exams (https://statistics.stanford.edu), accompanied by handwritten solutions that cover difficult proofs. The probability qualifying exam is held yearly and requires professional-level mathematical problem-solving. s1-teasers comprises 23 challenging brain-teasers commonly used in interview questions for quantitative trading positions. Each sample consists of a problem and solution taken from PuzzledQuant (https://www.puzzledquant.com/). We only take examples with the highest difficulty level (”Hard”).
为了补充这些现有数据集，我们创建了两个原始数据集。s1-prob 包含来自斯坦福大学统计学系博士资格考试（https://statistics.stanford.edu）概率部分的 182 个问题，附带涵盖困难证明的手写解答。概率资格考试每年举行一次，需要专业水平的数学问题解决能力。s1-teasers 包含 23 个常见的面试问题中的挑战性脑筋急转弯，这些脑筋急转弯常用于量化交易职位的面试问题。每个样本都包含来自 PuzzledQuant（https://www.puzzledquant.com/）的问题和解答。我们只选取难度最高的例子（“困难”）。

For each question, we generate a reasoning trace and solution using the Google Gemini Flash Thinking API (Google, 2024) extracting its reasoning trace and response. This yields 59K triplets of a question, generated reasoning trace, and generated solution. Examples from our dataset are in §D.2. We decontaminate all samples against our evaluation questions (MATH500, GPQA Diamond, AIME24; §C.5) using 8-grams and deduplicate the data.
对于每个问题，我们使用 Google Gemini Flash Thinking API（Google，2024）生成推理轨迹和解决方案，提取其推理轨迹和响应。这产生了 59K 个问题、生成推理轨迹和生成解决方案的三元组。我们的数据集示例见§D.2。我们使用 8-gram 对所有样本进行去污染，与我们的评估问题（MATH500、GPQA Diamond、AIME24；§C.5）进行对比，并去重数据。

We first remove any questions where we ran into any API errors reducing our dataset to 54,116 samples. Next, we filter out low-quality examples by checking if they contain any string patterns with formatting issues, such as ASCII art diagrams, non-existent image references, or inconsistent question numbering reducing our dataset to 51,581 examples. From this pool, we identify 384 samples for our final 1,000 samples from datasets that we perceive as high-quality and not in need of further filtering (see §C.4 for details).
我们首先删除了遇到任何 API 错误的问题，将数据集减少到 54,116 个样本。接下来，我们通过检查是否存在任何格式问题（如 ASCII 艺术图表、不存在的图像引用或不一致的问题编号）的字符串模式来过滤掉低质量示例，将数据集减少到 51,581 个示例。从这个池子中，我们确定了 384 个样本，作为最终 1,000 个样本，这些样本来自我们认为高质量且不需要进一步过滤的数据集（详情见§C.4）。

For difficulty, we use two indicators: model performance and reasoning trace length. We evaluate two models on each question: Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct (Qwen et al., 2024), with correctness assessed by Claude 3.5 Sonnet comparing each attempt against the reference solution (see §C.3 for the grading protocol). We measure the token length of each reasoning trace to indicate problem difficulty using the Qwen2.5 tokenizer. This relies on the assumption that more difficult problems require more thinking tokens. Based on the grading, we remove questions that either Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct can solve correctly and thus may be too easy. By using two models we reduce the likelihood of an easy sample slipping through our filtering due to a rare mistake on an easy question of one of the models. This brings our total samples down to 24,496, setting the stage for the next round of subsampling based on diversity. While filtering with these two models may be optimized for our setup as we will also use Qwen2.5-32B-Instruct as our model to finetune, the idea of model-based filtering generalizes to other setups.
对于难度，我们使用两个指标：模型性能和推理轨迹长度。我们对每个问题评估两个模型：Qwen2.5-7B-Instruct 和 Qwen2.5-32B-Instruct（Qwen 等人，2024 年），通过 Claude 3.5 Sonnet 比较每个尝试与参考解决方案的正确性来评估（见§C.3 以了解评分协议）。我们使用 Qwen2.5 分词器测量每个推理轨迹的标记长度，以指示问题难度。这基于一个假设，即更困难的问题需要更多的思考标记。基于评分，我们移除 Qwen2.5-7B-Instruct 或 Qwen2.5-32B-Instruct 可以正确解决的题目，因此可能过于简单。通过使用两个模型，我们减少了由于模型在简单问题上的罕见错误而使简单样本通过过滤的可能性。这使得我们的总样本量降至 24,496，为基于多样性的下一轮子采样奠定了基础。虽然使用这两个模型进行过滤可能针对我们的设置进行了优化，因为我们还将使用 Qwen2.5-32B-Instruct 作为我们的模型进行微调，但基于模型的过滤思想可以推广到其他设置。

To quantify diversity, we classify questions into domains using Claude 3.5 Sonnet based on the Mathematics Subject Classification (MSC) system (e.g., geometry, combinatorics, etc.) from the American Mathematical Society.¹¹1https://mathscinet.ams.org/mathscinet/msc/msc2020.html The taxonomy focuses on topics in mathematics but also includes other sciences such as biology, physics, and economics. To select our final examples from the pool of 24,496 questions, we first choose one domain uniformly at random. Then, we sample one problem from this domain according to a distribution that favors longer reasoning traces (see §C.4 for details) as motivated in Difficulty. We repeat this process until we have 1,000 total samples spanning 50 domains.
为了量化多样性，我们使用 Claude 3.5 Sonnet 根据美国数学学会的数学主题分类（MSC）系统（例如，几何学、组合数学等）将问题分类到各个领域。该分类法专注于数学领域，但也包括其他科学，如生物学、物理学和经济学。为了从 24,496 个问题的池中选出我们的最终示例，我们首先随机选择一个领域。然后，根据在难度中提出的、有利于更长的推理轨迹的分布，从这个领域抽取一个问题（详情见§C.4）。我们重复此过程，直到我们拥有跨越 50 个领域的 1,000 个总样本。

In §5.1, we will show that using our three criteria in combination is important, as only relying on quality, diversity, or difficulty in isolation leads to worse datasets. Some distilled generations are incorrect, which we allow in our data as we focus on capturing the reasoning process rather than entirely correct solutions. Our grader (§C.3) deems 53.6% correct in s1K and 63.0% in our follow-up s1K-1.1 (see §A).
在§5.1 中，我们将展示将我们的三个标准结合使用的重要性，因为仅依赖于质量、多样性或难度中的任何一个都会导致数据集质量下降。一些提炼出的生成内容是错误的，我们允许这些错误存在于我们的数据中，因为我们更关注捕捉推理过程而不是完全正确的解决方案。我们的评分者（§C.3）认为 s1K 中有 53.6%是正确的，在后续的 s1K-1.1 中有 63.0%（见§A）。

We propose a simple decoding-time intervention by forcing a maximum and/or minimum number of thinking tokens. Specifically, we enforce a maximum token count by simply appending the end-of-thinking token delimiter and optionally “Final Answer:” to early exit the thinking stage and make the model provide its current best answer. To enforce a minimum, we suppress the generation of the end-of-thinking token delimiter and optionally append the string “Wait” to the model’s current reasoning trace to encourage the model to reflect on its current generation. Figure 3 contains an example of how this simple approach can lead the model to arrive at a better answer.
我们提出了一种简单的解码时间干预方法，通过强制设置最大和/或最小数量的思考标记。具体来说，我们通过简单地添加思考结束标记并可选地添加“最终答案：”来提前退出思考阶段，使模型提供其当前最佳答案，从而强制执行最大标记计数。为了强制执行最小数量，我们抑制生成思考结束标记，并可选地添加字符串“等待”到模型的当前推理轨迹中，以鼓励模型反思其当前生成的内容。图 3 展示了这种简单方法如何引导模型得出更好的答案。

We benchmark budget forcing with: (I) Conditional length-control methods, which rely on telling the model in the prompt how long it should generate for. We group them by granularity into (a) Token-conditional control: We specify an upper bound of thinking tokens in the prompt; (b) Step-conditional control: We specify an upper bound of thinking steps, where each step is around 100 tokens; (c) Class-conditional control: We write two generic prompts that tell the model to either think for a short or long amount of time (see §E.1 for details). (II) Rejection sampling, which samples until a generation fits a predetermined compute budget. This oracle captures the posterior over responses conditioned on its length.
我们使用以下方法对预算强制进行基准测试：（I）条件长度控制方法，这些方法依赖于在提示中告诉模型它应该生成多长时间。我们将它们按粒度分为：（a）标记条件控制：我们在提示中指定思考标记的上限；（b）步骤条件控制：我们指定思考步骤的上限，每个步骤大约有 100 个标记；（c）类别条件控制：我们编写两个通用提示，告诉模型要么思考一段时间，要么思考很长时间（详情见§E.1）。（II）拒绝采样，它采样直到生成的内容符合预定的计算预算。这个预言家捕捉了基于其长度的响应的后验分布。

We perform supervised finetuning on Qwen2.5-32B-Instruct using s1K to obtain our model s1-32B using basic hyperparameters outlined in §D. Finetuning took 26 minutes on 16 NVIDIA H100 GPUs with PyTorch FSDP.
我们在 Qwen2.5-32B-Instruct 上使用 s1K 进行监督微调，以获得我们的 s1-32B 模型，该模型采用§D 中概述的基本超参数。微调在 16 个 NVIDIA H100 GPU 上使用 PyTorch FSDP 耗时 26 分钟。

We select three representative reasoning benchmarks widely used in the field: AIME24 (of America, 2024) has 30 problems that were used in the 2024 American Invitational Mathematics Examination (AIME) held from January 31 – February 1, 2024. AIME tests mathematical problem-solving with arithmetic, algebra, counting, geometry, number theory, probability, and other secondary school math topics. High-scoring high school students in the test are invited to participate in the United States of America Mathematics Olympiad (USAMO). All AIME answers are integers ranging from $000$ to $999$, inclusive. Some AIME problems rely on figures that we provide to our model using the vector graphics language Asymptote as it cannot take image inputs. MATH500 (Hendrycks et al., 2021) is a benchmark of competition math problems of varying difficulty. We evaluate on the same 500 samples selected by OpenAI in prior work (Lightman et al., 2023). GPQA Diamond (Rein et al., 2023) consists of 198 PhD-level science questions from Biology, Chemistry and Physics. Experts with PhDs in the corresponding domains only achieved 69.7% on GPQA Diamond (OpenAI, 2024). When we write “GPQA” in the context of evaluation in this work, we always refer to the Diamond subset. We build on the “lm-evaluation-harness” framework (Gao et al., 2021; Biderman et al., 2024). Unless otherwise specified, we evaluate with a temperature of 0 (greedy) and measure accuracy (equivalent to pass@1).
我们选择了三个在领域内广泛使用的代表性推理基准：AIME24（美国，2024）包含 30 个问题，这些问题用于 2024 年 1 月 31 日至 2 月 1 日举行的美国邀请数学竞赛（AIME）。AIME 通过算术、代数、计数、几何、数论、概率和其他中学数学主题来测试数学问题解决能力。在测试中得分较高的高中生被邀请参加美国数学奥林匹克（USAMO）。所有 AIME 答案都是整数，范围从 $000$ 到 $999$ ，包括 $000$ 和 $999$ 。一些 AIME 问题依赖于我们使用向量图形语言 Asymptote 提供的图形，因为我们的模型不能接受图像输入。MATH500（Hendrycks 等人，2021）是一个不同难度的竞赛数学问题基准。我们在 OpenAI 先前工作中选择的相同 500 个样本上进行评估（Lightman 等人，2023）。GPQA Diamond（Rein 等人，2023）由来自生物学、化学和物理的 198 个博士水平科学问题组成。只有对应领域的博士专家在 GPQA Diamond 上达到了 69.7%（OpenAI，2024）。 当我们在这项工作中提到“GPQA”时，我们总是指代 Diamond 子集。我们基于“lm-evaluation-harness”框架（高翔等，2021；比德曼等，2024）进行构建。除非另有说明，我们以 0（贪婪）的温度进行评估，并测量准确率（相当于 pass@1）。

We benchmark s1-32B against: OpenAI o1 series (OpenAI, 2024), closed-source models that popularized test-time scaling; DeepSeek r1 series (DeepSeek-AI et al., 2025), open-weight reasoning models with up to o1-level performance; Qwen’s QwQ-32B-preview (Team, 2024), a 32B open-weight reasoning model without disclosed methodology; Sky-T1-32B-Preview (Team, 2025) and Bespoke-32B (Labs, 2025), open models with open reasoning data distilled from QwQ-32B-preview and r1; Google Gemini 2.0 Flash Thinking Experimental (Google, 2024), the API that we distill from. As it has no official evaluation scores, we use the Gemini API to benchmark it ourselves. However, the “recitation error” of the Gemini API makes evaluation challenging.²²2https://github.com/google/generative-ai-docs/issues/257 We circumvent this, by manually inserting all 30 AIME24 questions in its web interface where the error does not appear. However, we leave out MATH500 (500 questions) and GPQA Diamond (198 questions), thus they are N.A. in LABEL:tab:perf. Our model, s1-32B, is fully open including weights, reasoning data, and code.
我们将 s1-32B 与以下基准模型进行比较：OpenAI o1 系列（OpenAI，2024），流行测试时缩放的闭源模型；DeepSeek r1 系列（DeepSeek-AI 等，2025），具有高达 o1 级性能的开源推理模型；Qwen 的 QwQ-32B-preview（团队，2024），一个 32B 开源推理模型，未公开方法；Sky-T1-32B-Preview（团队，2025）和 Bespoke-32B（实验室，2025），从 QwQ-32B-preview 和 r1 中提取开源推理数据的开放模型；Google Gemini 2.0 闪思实验（Google，2024），我们从其提炼的 API。由于它没有官方评估分数，我们使用 Gemini API 自行进行基准测试。然而，Gemini API 的“背诵错误”使得评估变得具有挑战性。通过手动将其所有 30 个 AIME24 问题插入其网页界面（错误不会出现），我们绕过了这一点。然而，我们排除了 MATH500（500 个问题）和 GPQA Diamond（198 个问题），因此它们在标签：tab：perf 中为 N.A.。我们的模型 s1-32B 完全开源，包括权重、推理数据和代码。

Figure 1 shows the performance of s1-32B with budget forcing scales with more test-time compute. In Figure 4 (left), we expand the plot from Figure 1 (middle) showing that while we can improve AIME24 performance using our budget forcing technique (§3) and more test-time compute it does eventually flatten out at six times. Suppressing the end-of-thinking token delimiter too often can lead the model into repetitive loops instead of continued reasoning. In Figure 4 (right), we show that after training Qwen2.5-32B-Instruct on our 1,000 samples to produce s1-32B and equipping it with the simple budget forcing technique, it operates in a different scaling paradigm. Scaling test-time compute on the base model via majority voting cannot catch up with the performance of s1-32B which validates our intuition from §3 that sequential scaling is more effective than parallel. We provide example generations of s1-32B in Figure 5.
图 1 展示了 s1-32B 在预算强制缩放和更多测试时间计算下的性能。在图 4（左）中，我们扩展了图 1（中）的图表，表明虽然我们可以通过我们的预算强制技术（§3）和更多测试时间计算来提高 AIME24 的性能，但它最终会在六倍时趋于平坦。过度抑制思考结束标记分隔符可能导致模型陷入重复循环而不是持续推理。在图 4（右）中，我们展示了在 1,000 个样本上训练 Qwen2.5-32B-Instruct 以产生 s1-32B，并配备简单的预算强制技术后，它运行在不同的缩放范式下。通过多数投票在基础模型上缩放测试时间计算无法赶上 s1-32B 的性能，这验证了我们在§3 中的直觉，即顺序缩放比并行缩放更有效。我们在图 5 中提供了 s1-32B 的示例生成。

In Figure 2 (right) and LABEL:tab:perf we compare s1-32B with other models. We find that s1-32B is the most sample-efficient open data reasoning model. It performs significantly better than our base model (Qwen2.5-32B-Instruct) despite just training it on an additional 1,000 samples. The concurrently released r1-32B shows stronger performance than s1-32B while also only using SFT (DeepSeek-AI et al., 2025). However, it is trained on 800 $\times$ more reasoning samples. It is an open question whether one can achieve their performance with just 1,000 samples. Finally, our model nearly matches Gemini 2.0 Thinking on AIME24. As the data for s1-32B is distilled from Gemini 2.0, this shows our distillation procedure was likely effective.
图 2（右）和标签：tab：perf 中，我们比较了 s1-32B 与其他模型。我们发现 s1-32B 是最具样本效率的开源数据推理模型。尽管只是在额外 1,000 个样本上进行了训练，它的表现显著优于我们的基础模型（Qwen2.5-32B-Instruct）。同时发布的 r1-32B 在 s1-32B 的基础上表现出更强的性能，同时仅使用了 SFT（DeepSeek-AI 等，2025）。然而，它是在 800 个 $\times$ 更多的推理样本上训练的。能否仅用 1,000 个样本达到其性能还是一个开放性问题。最后，我们的模型在 AIME24 上几乎与 Gemini 2.0 Thinking 相当。由于 s1-32B 的数据是从 Gemini 2.0 蒸馏而来，这表明我们的蒸馏过程可能是有效的。

In Table 3 we compare the test-time scaling methods we have introduced in §3. Overall, we find that budget forcing provides perfect control, good scaling, and leads to our best AIME24 score. Thus, this is the method we use for s1-32B in Figure 1 and in §4. In Table 4, we compare different strings for extrapolating performance. We find that “Wait” generally gives the best performance.
在表 3 中，我们比较了我们在第 3 节中介绍的计算时间缩放方法。总体而言，我们发现预算强制提供了完美的控制、良好的缩放，并导致了我们最佳的 AIME24 分数。因此，这是我们用于图 1 和第 4 节中 s1-32B 的方法。在表 4 中，我们比较了不同的字符串以预测性能。我们发现“等待”通常提供最佳性能。

Class-conditional control We provide benchmark scores for this method in §E.1 and summarize three findings here: (1) Token-conditional control fails without budget forcing, as our model cannot reliably count tokens - even when trained to do so. (2) Under step-conditional control, the model generates a similar total number of tokens when given different step targets, as the model goes from few steps with many tokens per step, to many steps with few tokens in each step. Thus, the model learns to hack its way around the compute constraint making the controllability of this method mediocre. (3) Class-conditional control can work - telling a model to simply think longer can increase its test-time compute and performance, which leads good scaling in Table 3.
类条件控制 我们在第 E.1 节中提供了此方法的基准分数，并在此总结三个发现：（1）在没有预算强制的情况下，标记条件控制失败，因为我们的模型无法可靠地计数标记——即使经过训练也是如此。（2）在步骤条件控制下，模型在给定不同的步骤目标时生成相似的总标记数，因为模型从每步许多标记的少量步骤，转变为每步少量标记的许多步骤。因此，模型学会了通过绕过计算约束来“黑客”其方式，使得此方法的可控性平庸。（3）类条件控制可以工作——告诉模型简单地思考更长可以增加其测试时的计算能力和性能，这在表 3 中导致了良好的扩展。

Surprisingly, we find that simply sampling until the generation fits a specific length leads to an inverse scaling trend as depicted in Figure 6. In §E.2 we inspect a question, which was answered correctly by the model when rejection sampling for $\leq 4000$, but not for the $\leq 8000$ token setting. In the $\leq 4000$ setting the model directly jumps to the correct approach, while for the $\leq 8000$ setting it backtracks a lot. We hypothesize that there is a correlation such that shorter generations tend to be the ones where the model was on the right track from the start, whereas longer ones tend to be ones where the model made mistakes and thus backtracks or questions itself. This leads to longer samples often being wrong when rejection sampling and thus the inverse scaling trend.
令人惊讶的是，我们发现简单地采样直到生成文本符合特定长度，会导致如图 6 所示的逆比例趋势。在§E.2 中，我们检查了一个问题，当对 $\leq 4000$ 进行拒绝采样时，模型正确回答了这个问题，但在 $\leq 8000$ 标记设置下没有。在 $\leq 4000$ 设置中，模型直接跳到正确的方法，而在 $\leq 8000$ 设置中，它后退了很多。我们假设存在一种相关性，即较短的生成文本往往是在模型从一开始就正确的情况下产生的，而较长的生成文本往往是在模型犯错误并因此回溯或质疑自己的情况下产生的。这导致在拒绝采样时，较长的样本往往是不正确的，从而产生了逆比例趋势。

There are a number of concurrent efforts to build models that replicate the performance of o1 (OpenAI, 2024). For example, DeepSeek-r1 and k1.5 (DeepSeek-AI et al., 2025; Team et al., 2025) are built with reinforcement learning methods, while others rely on SFT using tens of thousands of distilled examples (Team, 2025; Xu et al., 2025; Labs, 2025). We show that SFT on only 1,000 examples suffices to build a competitive reasoning model matching o1-preview and produces a model that lies on the pareto frontier (Figure 2). Further, we introduce budget forcing which combined with our reasoning model leads to the first reproduction of OpenAI’s test-time scaling curves (OpenAI, 2024). Why does supervised finetuning on just 1,000 samples lead to such performance gains? We hypothesize that the model is already exposed to large amounts of reasoning data during pretraining which spans trillions of tokens. Thus, the ability to perform reasoning is already present in our model. Our sample-efficient finetuning stage just activates it and we scale it further at test time with budget forcing. This is similar to the ”Superficial Alignment Hypothesis” presented in LIMA (Zhou et al., 2023), where the authors find that 1,000 examples can be sufficient to align a model to adhere to user preferences.
存在许多同时进行的努力，旨在构建能够复制 o1（OpenAI，2024）性能的模型。例如，DeepSeek-r1 和 k1.5（DeepSeek-AI 等，2025；团队等，2025）是使用强化学习方法构建的，而其他模型则依赖于使用数万个蒸馏示例的 SFT（Team，2025；徐等，2025；实验室，2025）。我们表明，在仅 1,000 个示例上进行 SFT 就足以构建一个与 o1-preview 相匹配的竞争性推理模型，并产生一个位于帕累托前沿的模型（图 2）。进一步地，我们引入了预算强制，它与我们的推理模型相结合，导致了 OpenAI 测试时缩放曲线的第一次复制（OpenAI，2024）。为什么仅在 1,000 个样本上进行监督微调就能带来如此大的性能提升？我们假设，在预训练期间，模型已经接触到了大量跨越万亿个标记的推理数据。因此，执行推理的能力已经存在于我们的模型中。我们的样本高效微调阶段只是激活了它，并在测试时通过预算强制进一步扩展它。 这与 LIMA（周等，2023）中提出的“表面对齐假设”类似，其中作者发现 1000 个示例就足以将模型对齐以符合用户偏好。

To evaluate and push the limits of these models, increasingly challenging benchmarks have been introduced, such as Olympiad-level science competitions (He et al., 2024; Jain et al., 2024; Zhong et al., 2023) and others (Srivastava et al., 2023; Glazer et al., 2024; Su et al., 2024; Kim et al., 2024; Phan et al., 2025). To enhance models’ performance on reasoning-related tasks, researchers have pursued several strategies: Prior works have explored continuing training language models on specialized corpora related to mathematics and science (Azerbayev et al., 2023; Yang et al., 2024), sometimes even synthetically generated data (Yu et al., 2024). Others have developed training methodologies specifically aimed at reasoning performance (Zelikman et al., 2022, 2024; Luo et al., 2025; Yuan et al., 2025; Wu et al., 2024a). Another significant line of work focuses on prompting-based methods to elicit and improve reasoning abilities, including methods like Chain-of-Thought prompting (Wei et al., 2023; Yao et al., 2023a, b; Bi et al., 2023; Fu et al., 2023; Zhang et al., 2024b; Xiang et al., 2025; Hu et al., 2024; Diao et al., 2024). These combined efforts aim to advance the reasoning ability of language models, enabling them to handle more complex and abstract tasks effectively.
为了评估和推动这些模型的极限，引入了越来越具有挑战性的基准，例如奥林匹克级别的科学竞赛（He 等，2024；Jain 等，2024；Zhong 等，2023）以及其他（Srivastava 等，2023；Glazer 等，2024；Su 等，2024；Kim 等，2024；Phan 等，2025）。为了提高模型在推理相关任务上的性能，研究人员采取了多种策略：先前的研究探讨了在数学和科学相关的专业语料库上继续训练语言模型（Azerbayev 等，2023；Yang 等，2024），有时甚至是合成的数据（Yu 等，2024）。其他人开发了专门针对推理性能的训练方法（Zelikman 等，2022，2024；Luo 等，2025；Yuan 等，2025；Wu 等，2024a）。另一项重要的工作线集中在基于提示的方法来激发和改进推理能力，包括如思维链提示（Wei 等，2023；Yao 等，2023a，b；Bi 等，2023；Fu 等，2023；Zhang 等，2024b；Xiang 等，2025；Hu 等，2024；Diao 等，2024）等方法。 这些联合努力旨在提升语言模型的推理能力，使它们能够有效地处理更复杂和抽象的任务。

As we introduce in §3, we differentiate two methods to scale test-time compute: parallel and sequential. The former relies on multiple solution attempts generated in parallel and selecting the best outcome via specific criteria. These criteria include choosing the most frequent response for majority voting or the best response based on an external reward for Best-of-N (Brown et al., 2024; Irvine et al., 2023; Levi, 2024). Unlike repeated sampling, previous sequential scaling methods let the model generate solution attempts sequentially based on previous attempts, allowing it to refine each attempt based on previous outcomes (Snell et al., 2024; Hou et al., 2025; Lee et al., 2025). Tree-based search methods (Gandhi et al., 2024; Wu et al., 2024b) offer a hybrid approach between sequential and parallel scaling, such as Monte-Carlo Tree Search (MCTS) (Liu et al., 2024; Zhang et al., 2023; Zhou et al., 2024; Choi et al., 2023) and guided beam search (Xie et al., 2023). REBASE (Wu et al., 2024b) employs a process reward model to balance exploitation and pruning during tree search. Empirically, REBASE has been shown to outperform sampling-based methods and MCTS (Wu et al., 2024b). Reward models (Lightman et al., 2023; Wang et al., 2024b, c) play a key role in these methods. They come in two variants: outcome reward models and process reward models. Outcome reward models (Xin et al., 2024; Ankner et al., 2024) assign a score to complete solutions and are particularly useful in Best-of-N selection, while process reward models (Lightman et al., 2023; Wang et al., 2024b; Wu et al., 2024b) assess individual reasoning steps and are effective in guiding tree-based search methods.
如我们在第 3 节中所述，我们区分了两种测试时计算缩放的两种方法：并行和顺序。前者依赖于并行生成的多个解决方案尝试，并通过特定标准选择最佳结果。这些标准包括选择最频繁的响应进行多数投票或基于外部奖励的最佳响应（Brown 等人，2024；Irvine 等人，2023；Levi，2024）。与重复抽样不同，之前的顺序缩放方法允许模型根据之前的尝试顺序生成解决方案尝试，从而根据之前的成果细化每个尝试（Snell 等人，2024；Hou 等人，2025；Lee 等人，2025）。基于树的方法（Gandhi 等人，2024；Wu 等人，2024b）提供了顺序和并行缩放之间的混合方法，例如蒙特卡洛树搜索（MCTS）（Liu 等人，2024；Zhang 等人，2023；Zhou 等人，2024；Choi 等人，2023）和引导束搜索（Xie 等人，2023）。REBASE（Wu 等人，2024b）在树搜索过程中采用过程奖励模型来平衡利用和剪枝。经验上，REBASE 已被证明优于基于采样的方法和 MCTS（Wu 等人，2024b）。 奖励模型（Lightman 等人，2023；王等人，2024b, c）在这些方法中发挥着关键作用。它们有两种变体：结果奖励模型和过程奖励模型。结果奖励模型（辛等人，2024；安克纳等人，2024）为完整解决方案分配分数，在最佳选择 N 中特别有用，而过程奖励模型（Lightman 等人，2023；王等人，2024b；吴等人，2024b）评估单个推理步骤，在指导基于树的搜索方法中非常有效。

We have shown that budget forcing allows extrapolating test-time compute in §4, e.g., improving AIME24 performance from 50% to 57%. However, it has two key limitations when scaling further: it eventually flattens out (Figure 4), and the context window of the underlying language model constrains it. Despite these constraints, our work shows test-time scaling across a wide range of accuracies (Figure 1), partly because scaling down test-time compute behaves predictably and does not suffer from these constraints.
我们已经证明了预算强制可以在第 4 节中扩展测试时的计算，例如，将 AIME24 的性能从 50%提高到 57%。然而，当进一步扩展时，它有两个关键限制：最终会趋于平稳（见图 4），并且底层语言模型的上下文窗口对其进行了限制。尽管存在这些限制，我们的工作表明在广泛的准确性范围内实现了测试时的扩展（见图 1），部分原因是因为降低测试时的计算行为具有可预测性，并且不受这些限制的影响。

Continuing test-time scaling will require approaches that can further extrapolate test-time compute. How can we get such extrapolation? There may be improvements to budget forcing such as rotating through different strings, not only “Wait”, or combining it with frequency penalties or higher temperature to avoid repetitive loops. An exciting direction for future work is also researching whether applying budget forcing to a reasoning model trained with reinforcement learning yields better extrapolation; or if RL allows for new ways of test-time scaling beyond budget forcing. Our work defines the right metrics (§3.2) – Control, Scaling, and Performance – to enable future research and progress on extrapolating test-time compute.
持续进行测试时缩放需要能够进一步外推测试时计算的方法。我们如何获得这种外推？可能可以通过旋转不同的字符串来改进预算强制，而不仅仅是“等待”，或者将其与频率惩罚或更高温度结合，以避免重复循环。未来工作的一个令人兴奋的方向是研究将预算强制应用于用强化学习训练的推理模型是否会产生更好的外推；或者 RL 是否允许在预算强制之外的新方法进行测试时缩放。我们的工作定义了正确的指标（§3.2）——控制、缩放和性能——以使未来的研究在测试时计算外推方面取得进展。

Parallel scaling offers one solution to the limits of sequential scaling, thus we augment our sequentially scaled model with two methods: (I) Majority voting: After generating $k$ solutions, the final solution is the most frequent one across generations; (II) Tree search via REBASE: We use the REBASE process reward model, which is initialized from LLaMA-34B and further finetuned on a synthetic process reward modeling dataset (Wu et al., 2024b). We then aggregate the solutions generated by REBASE via majority voting. As shown in Figure 7, augmenting our model with REBASE scales better than majority voting, and even sequential scaling in this scenario. However, REBASE requires an additional forward pass at each step for the reward model adding some computation overhead. For sequential scaling, when prompted to use up to 512 steps, for 12 out of the 30 evaluation questions the model generates a response that exceeds the context window leading to a large performance drop. Overall, we find that these parallel scaling methods complement sequential scaling thus they offer an avenue for scaling test-time compute even further; beyond fixed context windows.
并行扩展提供了一种解决顺序扩展限制的方法，因此我们通过两种方法增强我们的顺序扩展模型：（I）多数投票：在生成 $k$ 解决方案后，最终解决方案是跨代中最频繁的一个；（II）通过 REBASE 进行树搜索：我们使用 REBASE 过程奖励模型，该模型从 LLaMA-34B 初始化，并在合成过程奖励建模数据集（Wu 等人，2024b）上进行进一步微调。然后我们通过多数投票聚合 REBASE 生成的解决方案。如图 7 所示，将 REBASE 添加到我们的模型中比多数投票和在此场景中的顺序扩展具有更好的扩展性。然而，REBASE 需要在每个步骤进行额外的正向传递以添加奖励模型，这增加了计算开销。对于顺序扩展，当被提示使用最多 512 个步骤时，对于 30 个评估问题中的 12 个，模型生成的响应超出了上下文窗口，导致性能大幅下降。总的来说，我们发现这些并行扩展方法补充了顺序扩展，因此它们为扩展测试时计算提供了途径；超越固定上下文窗口。