Capabilities#  能力 #

To determine which model to use as the basis for continued pretraining, we test both multilingually-oriented models, and models that are targeted primarily toward English. We find that the strongest predictor of the model’s eventual capability in the target language is the model’s existing capability in English.
为确定持续预训练的基座模型，我们同时测试了多语言导向模型和主要面向英语的模型。研究发现，模型在目标语言最终能力的最强预测指标，是其现有的英语能力水平。

Our early experiments, illustrated in the figure below, demonstrate this: stronger English models like Llama 3.1 8B consistently adapted better to Finnish during CPT than multipurpose multilingual models that were less capable in English.
我们早期的实验（如下图所示）证明了这一点：像 Llama 3.1 8B 这样更强的英语模型在持续预训练(CPT)过程中对芬兰语的适应能力始终优于那些英语能力较弱的多用途多语言模型。

License#  许可证 #

Licensing is a major consideration when selecting a model to work with. Fully open source licenses like Apache 2.0 or MIT have the fewest restrictions on what you can do with the model after you train it. Other open weights models like Gemma or Llama models come with licensing constraints that may restrict what you can do with the models or their outputs.
选择模型时，许可证是主要考量因素。完全开源的许可证（如 Apache 2.0 或 MIT）对训练后模型使用的限制最少。其他开放权重的模型（如 Gemma 或 Llama 模型）则附带许可证约束，可能会限制您对模型或其输出的使用方式。

Size#  模型规模 #

Though they are more expensive to train and run, generally a larger model is best to maximize capabilities. Larger models tend to be more capable, more sample efficient, and are better at incorporating new training without damaging existing performance in other domains.
尽管训练和运行成本更高，但通常更大的模型能最大化性能表现。大模型往往能力更强、样本效率更高，且能更好地吸收新训练数据而不损害其他领域的现有表现。

In our early experiments, we did not include math data in the pretraining data mix (50B tokens, 70% Finnish, 25% English, 4% code, 1% paired texts.) Typically, when a domain is excluded during continued pretraining, model capabilities in that domain will decline, and that certainly occurred with the 8B model: math performance declined substantially. But as shown in the figure below, despite using the same data mix, math performance of the 70B model declined significantly less.
在早期实验中，我们的预训练数据混合集（500 亿 token，70%芬兰语，25%英语，4%代码，1%配对文本）未包含数学数据。通常情况下，当持续预训练中排除某个领域时，模型在该领域的能力会下降——80 亿参数模型确实出现了数学性能大幅衰退的现象。但如下图所示，尽管采用相同的数据混合比例，700 亿参数模型的数学性能下降幅度明显更小。

Other Considerations#  其他考量因素 #

Tokenizer efficiency in the target language is an important consideration when creating or selecting language models. Some models will have better tokenization for your target language than others. Tokenizer efficiency is usually measured with a metric called fertility–a measure of how many tokens it takes to represent a word, on average.
在创建或选择语言模型时，目标语言的分词器效率是一个重要考量因素。不同模型对目标语言的分词效果存在差异。分词器效率通常通过"生育率"指标来衡量——该指标反映了平均需要多少 token 才能表示一个单词。

Having an efficient tokenizer is not mandatory to achieve good results from continued pretraining, but inefficient tokenization means that the model needs to use more compute to generate the same amount of text, and less content can fit within the model’s context window. An inefficient tokenizer may also slightly reduce achievable model performance.
拥有高效的 tokenizer 对于持续预训练取得良好效果并非必需，但低效的分词意味着模型需要消耗更多计算资源来生成等量文本，且模型上下文窗口能容纳的内容更少。低效的 tokenizer 还可能轻微降低模型可达到的性能上限。

Sample scripts for measuring tokenizer fertility can be found in our evals repo. The following chart compares tokenizer fertility for several models, showing that while the base Llama 3.1 tokenizer is less efficient for Finnish than a custom one (like the original Poro), it is still a viable starting point.
测量分词器 fertility 的示例脚本可在我们的 evals 代码库中找到。下图对比了多个模型的分词器 fertility，表明虽然基础版 Llama 3.1 分词器对芬兰语的效率不如定制版本（如原版 Poro），但仍不失为一个可行的起点。

Model Selection Summary#
模型选择总结 #

Our analysis led us to select the Llama 3.1 8B and 70B models. The primary drivers for this decision were their excellent English performance, which we found to be a strong predictor of success in the target language, and the availability of both a large and small model within the same family.
经过分析，我们最终选择了 Llama 3.1 8B 和 70B 模型。这一决策的主要依据是：首先，它们在英语任务上的优异表现（我们研究发现这能有效预测目标语言的表现）；其次，该系列同时提供了大模型和小模型的选择。

Translation for Evaluations#
评估翻译方案 #

Translating existing English language evaluations into your target language is a practical option to quickly assess model performance in that language, but there are drawbacks:
将现有的英语评估翻译成目标语言是快速评估模型在该语言中表现的实用选择，但存在以下缺点：

• English based evaluations may contain embedded cultural assumptions that are not relevant or misleading in your target language.
  基于英语的评估可能包含与目标语言无关或具有误导性的文化假设

• Translation error can reduce the accuracy of the evaluation
  翻译错误会降低评估的准确性

However, we find that translated evals are still very useful, providing clear and useful signals to differentiate relative model performance. If you do not have the time or resources to create new evaluations from scratch, or pay for expensive human translations and annotations, machine translated evals are a perfectly acceptable way to begin.
但我们发现翻译后的评估仍然非常有用，能提供清晰有效的信号来区分模型的相对性能。若您没有时间或资源从头创建新评估，或支付昂贵的人工翻译和标注费用，机器翻译的评估是完全可接受的起步方案。

Previous published work [SB de Vroe et al.] supports the idea that machine translated evals, while imperfect, can still give useful information on model capabilities in the target language. Though it’s important to remember that the scores for the original and translated versions of an evaluation are unlikely to be directly comparable, due to the limitations mentioned above.
先前发表的研究[SB de Vroe 等]支持这一观点：机器翻译的评估虽不完美，仍能提供有关模型在目标语言中能力的有用信息。但需谨记，由于上述局限性，原始评估与其翻译版本的分数不太可能直接可比。

Translations for Post-Training Data#
翻译后训练数据的方法 #

While translation is a straightforward method of obtaining post-training datasets in another language, previous research has shown that translated data can introduce hallucinations into the model [Pipatanakul et al.] or are generally of lower quality than data natively-generated in the target language [Aryabumi et al., Dang et al.]. Generating post-training data from scratch in Finnish, however, means we will not be able to take advantage of the high-quality curated datasets available in English [Lambert et al.].
虽然翻译是获取另一种语言后训练数据集的直接方法，但先前研究表明，翻译数据可能向模型引入幻觉[Pipatanakul 等人]，或者通常质量低于目标语言原生生成的数据[Aryabumi 等人，Dang 等人]。然而，完全用芬兰语从头生成后训练数据，意味着我们将无法利用英语中现有的高质量精选数据集[Lambert 等人]。

To balance the need to minimise the use of translated data while ensuring that our model learns the same set of skills in English and Finnish, we translated only the prompts for the supervised fine-tuning (SFT) data and natively generated responses in Finnish, described below in the Synthetic Data section. The Direct Preference Optimization (DPO) data, however, needed to be translated with prompts and their corresponding chosen and rejected responses, though we did not ultimately use the translated versions.
为了在最小化翻译数据使用的同时确保模型能同等学习英语和芬兰语技能，我们仅翻译了监督微调(SFT)数据的提示词，并用芬兰语原生生成响应（如下文"合成数据"部分所述）。而直接偏好优化(DPO)数据则需要完整翻译提示词及其对应的选中/拒绝响应，不过我们最终并未使用这些翻译版本。

To translate post-training data, we use Poro, an earlier Finnish-English model we trained from scratch. In previous work, we benchmarked Poro and other MT models such as Opus-MT [Tiedemann et al.] and found that Poro offered better quality for Finnish translations among the open models [Luukkonen et al.]. As shown in the FLORES-101 benchmark results below, our Poro-34B model is competitive with other leading open translation systems.
为了翻译训练后数据，我们使用了 Poro——一个我们从头开始训练的早期芬兰语-英语模型。在先前的工作中，我们对 Poro 和其他机器翻译模型（如 Opus-MT [Tiedemann 等人]）进行了基准测试，发现 Poro 在开源模型中提供了更优质的芬兰语翻译[Luukkonen 等人]。如下方 FLORES-101 基准测试结果所示，我们的 Poro-34B 模型与其他领先的开源翻译系统相比具有竞争力。

Machine Translation Summary#
机器翻译总结 #

Our approach to translation was pragmatic and use-case driven. We invested in high-quality commercial translations for our evaluation benchmarks to ensure reliable measurement, and used our own open models for post-training data to balance cost and licensing. This hybrid strategy was essential for building a robust data pipeline, providing us with the translated benchmarks needed to properly evaluate our model’s new capabilities.
我们的翻译方法务实且以用例为导向。为确保评估基准的可靠测量，我们投资了高质量的商业翻译服务，同时采用自研的开源模型处理训练后数据以平衡成本与授权问题。这种混合策略对于构建稳健的数据管道至关重要，为我们提供了正确评估模型新能力所需的翻译基准。

Evaluation Frameworks#  评估框架 #

We use lm-evaluation-harness for most of our evaluations, and maintain our own fork with a number of translated evals for the languages we work with. (Another popular choice for evaluations is Lighteval.)
我们主要使用 lm-evaluation-harness 进行评估，并维护了自己的分支版本，其中包含针对我们涉及语言的多种翻译评估项。（另一个常用的评估工具是 Lighteval。）

We also maintain an automation framework to automate some of our eval workflow on Slurm-based HPC systems like LUMI, which reduces the manual effort in running a large number of evaluations.
我们还维护了一个自动化框架，用于在基于 Slurm 的 HPC 系统（如 LUMI）上自动化部分评估流程，这减少了运行大量评估所需的手动操作。

Pretraining Evaluations Selected#
选定的预训练评估项 #

We include a minimal set of popular, basic evaluations and their translated equivalents for our continued pretraining experiments. All of these are available in our lm-evaluation-harness fork. For our Finnish evaluations, we use existing translations of ARC, MMLU, HellaSwag, GSM8K, and TruthfulQA that were translated with DeepL [Thellmann et al].
在我们的持续预训练实验中，我们包含了一组最受欢迎的基础评估及其翻译版本。所有这些评估都可在我们的 lm-evaluation-harness 分支中找到。针对芬兰语评估，我们使用了由 DeepL 翻译的现有 ARC、MMLU、HellaSwag、GSM8K 和 TruthfulQA 翻译版本[Thellmann 等人研究]。

English  英语

• arc_challenge

• mmlu

• truthfulqa

• hellaswag

Finnish (translated)  芬兰语（已翻译）

• arc_challenge_mt_fi

• mmlu_mt_fi

• truthfulqa_mt_fi

• hellaswag_mt_fi

Code  代码

• humaneval

Math  数学

• gsm8k

• gsm8k_mt_fi

Translation  翻译

• flores200 (en to fi, fi to en)
  flores200（英语到芬兰语，芬兰语到英语）

Pretraining Evaluations Summary#
预训练评估总结

We established a multi-domain evaluation suite using the lm-evaluation-harness framework and a custom automation framework. The final set of benchmarks provided comprehensive coverage of the key capabilities we aimed to improve and preserve, forming the basis for the analysis in our experimental runs.
我们基于 lm-evaluation-harness 框架和定制自动化工具搭建了多领域评估套件。最终确定的基准测试集全面覆盖了需要提升和保持的核心能力维度，为实验运行分析提供了基础支撑。

Theory#  理论基础

Continued pretraining offers possibilities for compute and data efficient LLM training, utilizing knowledge transfer across domains. However, strong distribution shifts–such as training data in a new language–tend to result in catastrophic forgetting and capability loss on the original domains. Previous studies suggest that such forgetting can be alleviated by mixing the original training data with the new domain data [Ibrahim et al.]. For strong distribution shifts, the amount of the replayed original data required may even be 50% of the whole data mix, though there is a tradeoff: mixing in a lot of data from the original model domains may somewhat limit peak performance in the new domain, so finding the right balance between data in existing domains and data in new domains is key.
持续预训练通过跨领域知识迁移，为实现计算资源和数据高效的 LLM 训练提供了可能。然而，强烈的数据分布偏移（例如使用新语言训练数据）往往会导致原始领域的灾难性遗忘和能力丧失。已有研究表明[Ibrahim 等人]，通过将原始训练数据与新领域数据混合，可以缓解此类遗忘现象。面对显著的数据分布偏移时，重放的原始数据量甚至可能需要达到整体数据混合量的 50%，但这需要权衡取舍：混合过多原始模型领域的数据可能会在一定程度上限制新领域的峰值性能，因此找到现有领域数据与新领域数据之间的最佳平衡点至关重要。

For example, the figure below shows the impact of excluding math data during our continued pretraining experiments. In the first “no math” run, we used a 50B token data mix of 70% Finnish, 25% English, 4% code and 1% paired text, which caused math performance to decline in English, and remain low in Finnish. Swapping out the 4% code for 4% math substantially maintained English math performance during continued pretraining, while also boosting math performance in Finnish, despite the fact that math dataset is English oriented.
例如，下图展示了在持续预训练实验中排除数学数据的影响。在首次"无数学"实验中，我们采用了 500 亿 token 的数据混合比例（70%芬兰语、25%英语、4%代码和 1%配对文本），这导致英语数学性能下降，而芬兰语的数学能力仍保持较低水平。将 4%代码替换为 4%数学数据后，在持续预训练过程中基本维持了英语数学能力，同时显著提升了芬兰语的数学表现——尽管所用数学数据集是以英语为主的。

When applying continued pretraining on existing models, we rarely have access to the original training data and may even lack general information about what it contained, which increases the challenge of finding an optimal data mix. Because we are interested in maintaining performance in English, math and code while improving performance in Finnish, we combine data from all of these domains, and evaluate the impact of different data mixes.
在对现有模型进行持续预训练时，我们通常无法获取原始训练数据，甚至可能对其具体内容一无所知，这增加了寻找最优数据组合的难度。由于我们的目标是在保持英语、数学和代码领域性能的同时提升芬兰语表现，因此我们整合了来自所有这些领域的数据，并评估不同数据配比的影响。

Available Datasets#  可用数据集

Although data acquisition is a critical step in producing capable LLMs, it is also a challenging task including various data extraction, language identification, filtering and refinement steps. In this playbook we thus focus on the applicability of existing multilingual datasets. In particular we evaluate the Finnish subsets of FineWeb-2, HPLT v2 and CulturaX. To prevent catastrophic forgetting for English and code capabilities, our data mixes also included English, math, and code. To this end we utilize the FineWeb-Edu, StarCoder and FineMath datasets.
尽管数据采集是打造强大 LLMs 的关键步骤，但这也是一项包含数据提取、语言识别、筛选和精炼等多环节的复杂任务。因此在本实践指南中，我们重点关注现有多语言数据集的适用性。具体而言，我们评估了 FineWeb-2、HPLT v2 和 CulturaX 的芬兰语子集。为防止英语和编程能力的灾难性遗忘，我们的数据混合方案还包含英语、数学和编程内容。为此我们采用了 FineWeb-Edu、StarCoder 和 FineMath 数据集。

Continued Pretraining Data Mix Summary#
持续预训练数据混合方案总结 #

Our data strategy involved selecting high-quality, domain-specific datasets to cover all key capabilities. For Finnish, we evaluated several large web-based corpora. To preserve existing performance, we chose specialized datasets for English (FineWeb-Edu), code (StarCoder), and math (FineMath). The final proportions of these datasets in our training mix were determined through the experiments described in a later section.
我们的数据策略精选了高质量领域专用数据集以覆盖所有关键能力。针对芬兰语，我们评估了多个大型网络语料库。为保持现有性能，我们选择了英语专用数据集(FineWeb-Edu)、代码数据集(StarCoder)和数学数据集(FineMath)。这些数据集在训练混合中的最终比例将通过后文实验确定。

Pre-Tokenizing Training Data#
训练数据预分词处理 #

Pre-tokenizing converts training text from jsonl-formatted input files into numerical, tokenized representations. The data at this stage often consists of hundreds or thousands of files from different data sets. Preprocessing of the files is done in parallel and as a result for each of the input files we get 2 files following megatron mmap format (binary data file and index file). Example for converting a single file below
预分词将 jsonl 格式的训练文本文件转换为数值化的分词表示。此阶段的数据通常包含来自不同数据集的数百或数千个文件。文件预处理采用并行方式完成，最终为每个输入文件生成两个符合 megatron mmap 格式的文件（二进制数据文件和索引文件）。下方是转换单个文件的示例：

#!/bin/bash

# Paths (update these or pass the paths as a parameter)
INPUT_FILE=$1
OUTPUT_FOLDER=$2
OUTPUT_FILE=$OUTPUT_FOLDER/$(basename $INPUT_FILE)
WORKERS=...
mkdir -p $OUTPUT_FOLDER

python ./Megatron-LM/tools/preprocess_data.py \
    --input $INPUT_FILE \
    --output $OUTPUT_FILE \
    --json-keys text \
    --tokenizer-type HuggingFaceTokenizer \
    --tokenizer-model <path_to_your_hf_model> \
    --append-eod \
    --log-interval 50000 \
    --workers $WORKERS

The converted files are then merged into larger files, one per original dataset. These files follow the same mmap format (.bin, .idx). During training Megatron-LM creates batches of data at the desired data mix by specifying sampling priorities for each one of the merged dataset files.
转换后的文件随后会被合并为更大的文件，每个原始数据集对应一个合并文件。这些文件遵循相同的 mmap 格式（.bin 和.idx）。在训练过程中，Megatron-LM 通过为每个合并数据集文件指定采样优先级，按所需数据比例创建数据批次。

#!/bin/bash

# Paths (update these or pass them as a parameter)
# INPUT_FOLDER contains multiple .bin and .idx files which are combined
# OUTPUT_FILE is the name of the merged file without .bin or .idx suffix

INPUT_FOLDER=$1
OUTPUT_FOLDER=$2
OUTPUT_FILE=$3

mkdir -p $OUTPUT_FOLDER

python ./Megatron-LM/tools/merge_datasets.py \
    --input $INPUT_FOLDER \
    --output-prefix $OUTPUT_FOLDER/$OUTPUT_FILE

Reference Slurm scripts for tokenization and merging in Lumi environment are available in our scripts repository.
参考用于 Lumi 环境中分词与合并任务的 Slurm 脚本已发布于我们的脚本仓库。

Converting model to Megatron format#
将模型转换为 Megatron 格式

For training with Megatron-LM, we convert the base model from Hugging Face format into a megatron checkpoint. A decision of the parallelism configuration (Tensor parallel, Pipeline parallel) is made at this point. We used TP=2, PP=1 for 8B models and TP=8, PP=8 for the 70B model on the LUMI AMD instinct MI250X-based cluster. An example script for converting the 8B model to Megatron format below
在使用 Megatron-LM 进行训练时，我们需将 Hugging Face 格式的基础模型转换为 megatron 检查点格式。此时需要确定并行配置方案（张量并行、流水线并行）。在基于 LUMI AMD Instinct MI250X 集群上，我们对 8B 模型采用 TP=2、PP=1 配置，对 70B 模型采用 TP=8、PP=8 配置。下方展示了将 8B 模型转换为 Megatron 格式的示例脚本：

#!/bin/bash

HF_FORMAT_DIR=<path to your hf model>
TOKENIZER_MODEL=$HF_FORMAT_DIR
TARGET_PP=1
TARGET_TP=2
MEGATRON_FORMAT_DIR=megatron-checkpoints/llama3.1-8B-TP-$TARGET_TP-PP-$TARGET_PP

python3 Megatron-LM/tools/checkpoint/convert.py \
    --model-type GPT \
    --loader llama_mistral \
    --model-size llama3-8B \
    --checkpoint-type 'hf' \
    --saver mcore \
    --target-tensor-parallel-size ${TARGET_TP} \
    --target-pipeline-parallel-size ${TARGET_PP} \
    --load-dir ${HF_FORMAT_DIR} \
    --save-dir ${MEGATRON_FORMAT_DIR} \
    --tokenizer-model ${TOKENIZER_MODEL}

Limitation: Converting checkpoint to use Virtual pipeline parallel (VPP) was not working correctly at time of writing, but using VPP would give a performance benefit with the 70B model.
限制：在撰写本文时，将检查点转换为使用虚拟管道并行（VPP）尚未正常工作，但使用 VPP 将为 70B 模型带来性能优势。

Reference scripts for converting 8B and 70B Llama models to Megatron format in a Slurm-based environment like LUMI are available in our scripts repository.
用于在 LUMI 等基于 Slurm 的环境中，将 8B 和 70B LLaMA 模型转换为 Megatron 格式的参考脚本可在我们的脚本仓库中获取。

Training the model#  训练模型

Once we’ve converted the model to a Megatron-LM-compatible checkpoint, we are ready to begin training. Refer to our example Megatron-LM pretraining Slurm script in our scripts repository for the specifics of our Megatron configuration.
将模型转换为兼容 Megatron-LM 的检查点后，即可开始训练。具体配置参数请参考我们脚本库中的 Megatron-LM 预训练 Slurm 脚本示例。

Based on our testing and the reference configuration for the Llama 3.1 family models, these are the Hyperparameters we chose for training the model.
根据我们的测试及 Llama 3.1 系列模型的参考配置，以下是训练模型时选择的超参数。

Converting megatron checkpoint to Hugging Face transformers format#
将 Megatron 检查点转换为 Hugging Face transformers 格式 #

After the model is trained, the final Megatron checkpoint is converted back to Hugging Face transformers checkpoint for easier use on other platforms and downstream tasks. Here is an example script to convert the latest checkpoint in LOAD_DIR to Hugging Face transformers format.
模型训练完成后，最终的 Megatron 检查点会被转换回 Hugging Face transformers 检查点格式，以便在其他平台和下游任务中更便捷地使用。以下是将 LOAD_DIR 目录中最新的检查点转换为 Hugging Face transformers 格式的示例脚本。

#!/bin/bash

# paths (update these or pass as a parameter)
LOAD_DIR=$1
OUT_DIR=$2
TOKENIZER_DIR=...  # <----- Change this to your needs

python3 Megatron-LM/tools/checkpoint/convert.py \
--model-type GPT \
--loader mcore \
--saver llama_mistral \
--load-dir $LOAD_DIR \
--save-dir $OUT_DIR \
--tokenizer-dir $TOKENIZER_DIR \

We have an example script for converting any checkpoint (not just the latest) in load directory here LumiOpen/poro2-scripts-dev
我们在 LumiOpen/poro2-scripts-dev 中提供了转换加载目录中任意检查点（不仅限于最新）的示例脚本

Continued Pretraining with Megatron-LM Summary#
使用 Megatron-LM 进行持续预训练总结 #

The core technical workflow involved preparing data into the Megatron mmap format, converting Hugging Face checkpoints to the Megatron format with appropriate parallelism settings (TP/PP), and executing the training. The example scripts and hyperparameters provided in this section serve as a template for replicating this process on a similar HPC system.
核心技术流程包括：将数据准备成 Megatron mmap 格式，以适当的并行设置（TP/PP）将 Hugging Face 检查点转换为 Megatron 格式，并执行训练。本节提供的示例脚本和超参数可作为在类似 HPC 系统上复现此过程的模板。

Experimental Runs Summary#
实验运行总结

Through systematic experimentation, we determined our final, optimized configuration. The key findings were: 1) a higher learning rate (3E-04 for 8B) was effective; 2) repeating data did not improve performance; 3) FineWeb2 was the best-performing Finnish dataset; and 4) a balanced data mix of 30% Finnish, 30% English, 30% code, and 10% math was crucial for preventing catastrophic forgetting. This configuration, detailed in the table above, formed the basis for our final pretraining run.
通过系统性实验，我们确定了最终优化配置。关键发现包括：1）较高学习率（8B 模型采用 3E-04）效果显著；2）数据重复未提升性能；3）FineWeb2 是表现最佳的芬兰语数据集；4）30%芬兰语+30%英语+30%代码+10%数学的平衡数据配比能有效防止灾难性遗忘。上表详述的这一配置成为最终预训练的基础方案。

Poro 2 8B#  Poro 2 8B

For the 8B model our approach leads to a consistent improvement in all Finnish benchmarks, with ~10pp average improvement. We are also able to maintain most of the original English capabilities with a 1pp average decrease in our evaluations.
对于 8B 模型，我们的方法在所有芬兰语基准测试中均实现了持续改进，平均提升约 10 个百分点。同时，在评估中我们能够保持原始英语能力的绝大部分，仅平均下降 1 个百分点。

Although we observe almost 14pp improvement on the Finnish GSM8K math evaluation, we are not able to maintain the original Llama 3.1 performance on the English variant of GSM8K. If higher math performance is critical, adding more math data to the mix, or annealing with a higher mix of high quality math data would help improve performance.
尽管我们在芬兰语 GSM8K 数学评估中观察到近 14 个百分点的提升，但无法在英语版 GSM8K 上保持原始 Llama 3.1 的性能水平。若需提升数学能力表现，可通过增加数学训练数据比例，或采用更高质量数学数据进行退火训练来改善效果。

Poro 2 70B#

Llama 3.1 70B has considerably stronger initial Finnish capabilities than the smaller variant, but our continued pretraining approach still provides improvements in all of our Finnish benchmarks. The larger model is also able to maintain, and in most cases even improve, on its English capabilities. Overall Finnish scores improve by an average of ~4pp, while English capabilities remain largely the same with a ~0.4pp average improvement.
Llama 3.1 70B 较大型变体初始芬兰语能力显著更强，但我们的持续预训练方法仍能提升所有芬兰语基准测试表现。该大模型不仅能保持英语能力，多数情况下甚至有所提升。芬兰语总体得分平均提高约 4 个百分点，而英语能力基本持平，平均微升 0.4 个百分点。

Performance Trade-offs: Code and Math#
性能权衡：代码与数学 #

Our primary goal for this project was to significantly enhance Finnish language capabilities. The results demonstrate clear success on that front. However, this focus on language acquisition revealed the challenges of maintaining performance in specialized domains when the original training data is unknown.
本项目的主要目标是显著提升芬兰语能力。结果显示我们在这方面取得了明显成功。然而，这种对语言习得的专注也揭示了当原始训练数据未知时，在专业领域保持性能所面临的挑战。

On the humaneval benchmark, for instance, our Poro 2 models do not match the performance of the original Llama 3.1 models. This is a common challenge in continued pretraining. Without access to the original pretraining dataset, it is difficult to replicate the exact data quality and domain balance that gave the base model its initial capabilities. While our data mix included a substantial portion of code (30% from StarCoder), the performance gap suggests that the composition or quality of the original data was different from our own.
例如在 humaneval 基准测试中，我们的 Poro 2 模型性能未能达到原始 Llama 3.1 模型的水平。这是持续预训练中常见的挑战。由于无法获取原始预训练数据集，我们难以复现赋予基础模型初始能力的数据质量和领域平衡。虽然我们的数据混合包含了大量代码（30%来自 StarCoder），但性能差距表明原始数据的构成或质量与我们的存在差异。

This highlights a key principle of CPT: performance is highly sensitive to the data mix, and adapting a model often involves navigating trade-offs between new and existing capabilities, especially with incomplete knowledge of the original training recipe. For projects where coding or math proficiency is the highest priority, one might need to experiment with different data allocations or seek out more specialized, high-quality datasets for those domains.
这凸显了持续预训练（CPT）的一个关键原则：模型性能对数据组合极为敏感，而模型适配往往需要在新增能力与既有能力之间权衡取舍——尤其是在原始训练配方信息不完整的情况下。若项目以编程或数学能力为最高优先级，开发者可能需要尝试不同的数据配比方案，或专门为这些领域寻找更专业、更高质量的数据集。

Final Pretraining Results Summary#
最终预训练结果总结

The final results confirm that our CPT strategy was highly effective at its primary goal: we achieved significant gains in Finnish across all benchmarks. The analysis also highlights the critical nature of data mixing and model size in managing performance trade-offs. While the larger 70B model showed more resilience, the performance drops in specialized domains like coding and math underscore the challenges of CPT when the original training data is unknown. This demonstrates that CPT is a powerful method for language acquisition, provided that data mixes are carefully balanced to align with a project’s specific priorities.
最终结果证实，我们的持续预训练（CPT）策略在主要目标上成效显著：芬兰语在所有基准测试中均取得重大提升。分析同时揭示了数据混合比例与模型规模对性能平衡的关键影响。尽管 70B 大模型展现出更强鲁棒性，但在编程和数学等专业领域出现的性能下滑，凸显了原始训练数据未知时 CPT 面临的挑战。这表明只要精心平衡数据配比以契合项目优先级，CPT 就能成为语言习得的强效方法。

Post-training evaluation#
后训练评估 #

We evaluate our post-training checkpoints on general instruction following and open-ended conversational following in English and Finnish.
我们在英语和芬兰语的通用指令遵循及开放式对话任务上评估了后训练检查点的表现。

Post-training process#  训练后处理流程

Post-training is a process that trains a pretrained model to act as an assistant that can respond to user prompts and follow instructions. Our post-training pipeline involves one round of supervised fine-tuning (SFT) followed by preference tuning, specifically direct preference optimization (DPO). We use the Transformers Reinforcement Learning library (TRL) as our post-training framework.
后训练是一个对预训练模型进行再训练的过程，使其成为能够响应用户提示并遵循指令的助手。我们的后训练流程包括一轮监督微调（SFT），随后进行偏好调优，具体采用直接偏好优化（DPO）方法。我们使用 Transformers 强化学习库（TRL）作为后训练框架。

Our post-training codebase is a fork of the Alignment Handbook [Tunstall et al.]. We make our codebase and recipes available at: LumiOpen/alignment-handbook. Please refer to the alignment handbook repository for details on how to run the code.
我们的后训练代码库是基于 Alignment Handbook[Tunstall 等人]的一个分支。我们在 LumiOpen/alignment-handbook 提供了代码库和训练方案。具体运行方法请参考 alignment handbook 代码库。

Datasets available#  可用数据集

There are a large number of post-training datasets currently available with varying levels of quality and diversity, most of them in English. Our SFT data mixture is built around the prompts from the Tulu 3 SFT mix, which are targeted towards improving a model’s abilities on a large number of tasks. We combine prompts from this dataset with several others to create a diverse, bilingual instruction set.
目前已有大量不同质量和多样性的训练后数据集可供使用，其中大部分为英文数据。我们的监督微调(SFT)数据混合方案以 Tulu 3 SFT 混合集中的提示词为核心构建，这些提示词旨在提升模型在多种任务上的表现。我们将该数据集的提示词与其他多个来源相结合，最终构建出一个多样化的双语指令集。

Post-training Summary#  训练后总结 #

Our post-training pipeline was a multi-stage process designed to transform the CPT base models into helpful assistants. We established a new suite of chat and instruction-following evaluations for this phase. The process involved a full-parameter SFT on a large, custom-bilingual dataset, followed by DPO using the open-license HelpSteer3 dataset to further refine the models’ helpfulness and safety.
我们的训练后流程是一个多阶段设计，旨在将 CPT 基础模型转化为实用的助手。为此阶段我们建立了一套全新的对话和指令遵循评估体系。该流程包括在大型定制双语数据集上进行全参数监督微调(SFT)，随后使用开源许可的 HelpSteer3 数据集进行直接偏好优化(DPO)，以进一步提升模型的实用性和安全性。

SFT prompt selection#  SFT 提示选择

We did not use the full set of Tulu 3 prompts for our SFT phase. We filtered out the subset of prompts that came from non-commercially usable datasets, removed non-English prompts, and then deduplicated the remaining prompts. These selected prompts were used as the basis for our synthetic data work.
我们在 SFT 阶段并未使用完整的 Tulu 3 提示集。我们筛选掉了来自非商业可用数据集的提示子集，移除了非英语提示，并对剩余提示进行了去重处理。这些精选提示构成了我们合成数据工作的基础。

Inference Pipeline#  推理流水线 #

In order to do large scale inference on Slurm with batch jobs, we utilized our internally developed dispatcher framework. Dispatcher makes it easy to horizontally scale inference workloads, handles resuming work if a job ends before the task is complete, and avoids the need to pre-partition the data for workers.
为了在 Slurm 上通过批处理作业进行大规模推理，我们采用了内部开发的调度器框架。该调度器能轻松实现推理工作负载的水平扩展，处理作业未完成时的任务恢复，并避免了为工作节点预先分割数据的需要。

When generating potential model responses, we generated up to 64 parallel responses to each prompt, then selected a high quality response from among the possible options using a reward or judge model.
在生成潜在模型响应时，我们为每个提示最多生成 64 个并行响应，然后通过奖励模型或评判模型从候选选项中筛选出高质量响应。

Reward Models#  奖励模型

Multi-Turn Instruction Following#
多轮指令跟随

Using only single-turn instruction data during SFT results in a model that cannot follow multi-turn conversation. To supplement the multi-turn samples from Tulu 3, we generated more multi-turn conversations in English using the Magpie method [Xu et al.]. The key insight of Magpie is that when an instruction-tuned model is given the chat template for the user side of the conversation, but without any user query, the model will generate its own prompt because of the autoregressive nature of the model. The model can then respond to its self-generated prompt. A subsequent conversation turn can be generated by prepending the previous turn to the user template. We found, however, that the Llama models did not reliably generate well-formed conversations in this setting and that we needed a slightly more complex multi-step prompting approach to clean up the generated data.
在 SFT 阶段仅使用单轮指令数据会导致模型无法处理多轮对话。为补充 Tulu 3 中的多轮对话样本，我们采用 Magpie 方法[徐等人]生成了更多英文多轮对话。Magpie 的核心思路是：当给定用户侧对话模板但未提供具体查询时，经过指令调校的自回归模型会自主生成提示词并作出响应。通过将前轮对话附加到用户模板前，即可生成后续对话轮次。但我们发现 Llama 模型在此设定下无法稳定生成格式良好的对话，需要采用更复杂的多步提示策略来优化生成数据质量。

Synthetic Data Generation Summary#
合成数据生成总结 #

Our synthetic data pipeline was a critical enabler for post-training. By using a dispatcher for large-scale inference and leveraging reward models (ArmoRM for English, LLM-as-a-Judge for Finnish), we generated the necessary SFT data. While our fluency rating experiments for Finnish yielded limited results, the overall pipeline successfully produced the data needed to align the Poro 2 models.
我们的合成数据流水线是后续训练的关键推动因素。通过使用调度器进行大规模推理，并利用奖励模型（英语采用 ArmoRM，芬兰语采用 LLM-as-a-Judge），我们生成了必要的监督微调(SFT)数据。虽然针对芬兰语的流畅度评级实验成果有限，但整体流水线成功产出了对齐 Poro 2 模型所需的数据。

Complete 8B post-training results#
完整的 80 亿参数后训练结果

Complete 70B posttraining results#
70B 完整后训练结果#