DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-V2：强大、经济、高效的专家混合语言模型

(a) MMLU accuracy vs. activated parameters, among different (b) open-source models.
（a） MMLU 精度与激活参数，在不同的 （b） 开源模型中。

(b) Training costs and inference efficiency of DeepSeek 67B (Dense) and DeepSeek-V2.
（b） DeepSeek 67B （Dense） 和 DeepSeek-V2 的训练成本和推理效率。

1 Introduction ..... 4 1 引言 .....4
2 Architecture ..... 6
2 建筑.....6
2.1 Multi-Head Latent Attention: Boosting Inference Efficiency, ..... 6
2.1 多头潜在注意力：提高推理效率，.....6
2.1.1 Preliminaries: Standard Multi-Head Attention ..... 6
2.1.1 预赛：标准多头注意力.....6
2.1.2 Low-Rank Key-Value Joint Compression ..... 7
2.1.2 低秩键值关节压缩.....7
2.1.3 Decoupled Rotary Position Embedding ..... 7
2.1.3 解耦旋转位置嵌入.....7
2.1.4 Comparison of Key-Value Cache ..... 8
2.1.4 键值缓存.....比较8
2.2 DeepSeekMoE: Training Strong Models at Economical Costs ..... 9
2.2 DeepSeekMoE：以经济的成本训练强大的模型 .....9
2.2.1 Basic Architecture ..... 9
2.2.1 基本架构.....9
2.2.2 Device-Limited Routing . ..... 9
2.2.2 设备受限路由。.....9
2.2.3 Auxiliary Loss for Load Balance ..... 9
2.2.3 负载均衡.....的辅助损耗9
2.2.4 Token-Dropping Strategy ..... 11
2.2.4 代币投放策略.....11
3 Pre-Training ..... 11
3 预训练.....11
3.1 Experimental Setups ..... 11
3.1 实验设置.....11
3.1.1 Data Construction ..... 11
3.1.1 数据构建.....11
3.1.2 Hyper-Parameters ..... 11
3.1.2 超参数.....11
3.1.3 Infrastructures ..... 12
3.1.3 基础设施.....12
3.1.4 Long Context Extension ..... 12
3.1.4 长上下文扩展.....12
3.2 Evaluations ..... 13
3.2 评价.....13
3.2.1 Evaluation Benchmarks ..... 13
3.2.1 评估基准 .....13
3.2.2 Evaluation Results ..... 14
3.2.2 评价结果 .....14
3.2.3 Training and Inference Efficiency ..... 15
3.2.3 训练和推理效率.....15
4 Alignment ..... 16
4 对齐.....16
4.1 Supervised Fine-Tuning ..... 16
4.1 监督微调.....16
4.2 Reinforcement Learning ..... 16
4.2 强化学习.....16
4.3 Evaluation Results ..... 17
4.3 评估结果 .....17
4.4 Discussion ..... 19
4.4 讨论.....19
5 Conclusion, Limitation, and Future Work ..... 20
5 结论、局限性和未来工作.....20
A Contributions and Acknowledgments ..... 27
a 文稿和致谢.....27
B Full Formulas of MLA ..... 29
B MLA的完整公式.....29
C Ablation of Attention Mechanisms ..... 29
C 消融注意力机制 .....29
C. 1 Ablation of MHA, GQA, and MQA ..... 29
C. 1 MHA、GQA 和 MQA .....的消融29
C. 2 Comparison Between MLA and MHA ..... 29
C. 2 MLA与MHA.....的比较29
D Discussion About Pre-Training Data Debiasing ..... 30
D 关于预训练数据去偏.....的讨论30
E Additional Evaluations on Math and Code ..... 31
E 数学和代码.....的附加评估31
F Evaluation Formats ..... 32
F 评估格式 .....32

In the past few years, Large Language Models (LLMs) (Anthropic, 2023: Google, 2023 OpenAI, 2022, 2023) have undergone rapid development, offering a glimpse into the dawn of Artificial General Intelligence (AGI). In general, the intelligence of an LLM tends to improve as the number of parameters increases, allowing it to exhibit emergent capabilities across various tasks (Wei et al., 2022). However, the improvement comes at the cost of larger computing resources for training and a potential decrease in inference throughput. These constraints present significant challenges that impede the widespread adoption and utilization of LLMs. In order to tackle this problem, we introduce DeepSeek-V2, a strong open-source Mixture-of-Experts (MoE) language model, characterized by economical training and efficient inference through an innovative Transformer architecture. It is equipped with a total of 236B parameters, of which 21B are activated for each token, and supports a context length of tokens.
在过去的几年里，大型语言模型（LLMs）（Anthropic，2023：Google，2023 OpenAI，2022,2023）经历了快速发展，让我们得以一窥通用人工智能（AGI）的曙光。一般来说，随着参数数量的增加，一个人LLM的智力往往会提高，使其能够在各种任务中表现出紧急能力（Wei 等人，2022 年）。然而，这种改进是以更大的计算资源为代价的，并且可能会降低推理吞吐量。这些制约因素带来了重大挑战，阻碍了 LLMs.为了解决这个问题，我们引入了 DeepSeek-V2，这是一种强大的开源专家混合 （MoE） 语言模型，其特点是通过创新的 Transformer 架构进行经济的训练和高效的推理。它总共配备了 236B 参数，其中 21B 为每个令牌激活，并支持 令牌的上下文长度。

We optimize the attention modules and Feed-Forward Networks (FFNs) within the Transformer framework (Vaswani et al., 2017) with our proposed Multi-head Latent Attention (MLA) and DeepSeekMoE. (1) In the context of attention mechanisms, the Key-Value (KV) cache of the Multi-Head Attention (MHA) (Vaswani et al., 2017) poses a significant obstacle to the inference efficiency of LLMs. Various approaches have been explored to address this issue, including Grouped-Query Attention (GQA) (Ainslie et al. 2023 ) and Multi-Query Attention (MQA) (Shazeer 2019). However, these methods often compromise performance in their attempt to reduce the cache. In order to achieve the best of both worlds, we introduce MLA, an attention mechanism equipped with low-rank key-value joint compression. Empirically, MLA achieves superior performance compared with MHA, and meanwhile significantly reduces the KV cache during inference, thus boosting the inference efficiency. (2) For Feed-Forward Networks (FFNs), we follow the DeepSeekMoE architecture (Dai et al. 2024), which adopts fine-grained expert segmentation and shared expert isolation for higher potential in expert specialization. The DeepSeekMoE architecture demonstrates great advantages compared with conventional MoE architectures like GShard (Lepikhin et al., 2021), enabling us to train strong models at an economical cost. As we employ expert parallelism during training, we also devise supplementary mechanisms to control communication overheads and ensure load balance. By combining these two techniques, DeepSeek-V2 features strong performance (Figure 1(a)), economical training costs, and efficient inference throughput (Figure 1(b)), simultaneously.
我们用我们提出的多头潜在注意力（MLA）和DeepSeekMoE优化了Transformer框架（Vaswani等人，2017）中的注意力模块和前馈网络（FFN）。（1）在注意力机制的背景下，多头注意力（MHA）的键值（KV）缓存（Vaswani et al.， 2017）对推理LLMs效率构成了重大障碍。已经探索了各种方法来解决这个问题，包括分组查询注意力 （GQA） （Ainslie 等人，2023 年）和多查询注意力 （MQA） （Shazeer 2019）。但是，这些方法在尝试减少 缓存时通常会损害性能。为了实现两全其美，我们引入了MLA，这是一种配备低秩键值联合压缩的注意力机制。从经验上看，MLA与MHA相比性能优越，同时显著降低了推理过程中的KV缓存，从而提高了推理效率。（2）对于前馈网络（FFN），我们遵循DeepSeekMoE架构（Dai et al. 2024），该架构采用细粒度的专家分割和共享的专家隔离，以提高专家专业化的潜力。与 GShard 等传统 MoE 架构相比，DeepSeekMoE 架构表现出巨大的优势（Lepikhin 等人，2021 年），使我们能够以经济的成本训练强大的模型。当我们在培训期间采用专家并行性时，我们还设计了补充机制来控制通信开销并确保负载平衡。通过结合这两种技术，DeepSeek-V2 同时具有强大的性能（图 1（a））、经济的训练成本和高效的推理吞吐量（图 1（b））。

We construct a high-quality and multi-source pre-training corpus consisting of tokens. Compared with the corpus used in DeepSeek 67B (our previous release) (DeepSeek-AI |2024), this corpus features an extended amount of data, especially Chinese data, and higher data quality. We first pretrain DeepSeek-V2 on the full pre-training corpus. Then, we collect conversational sessions, which encompass various domains such as math, code, writing, reasoning, safety, and more, to perform Supervised Fine-Tuning (SFT) for DeepSeek-V2 Chat (SFT). Finally, we follow DeepSeekMath (Shao et al., 2024) to employ Group Relative Policy Optimization (GRPO) to further align the model with human preference and produce DeepSeek-V2 Chat (RL).
我们构建了一个由 代币组成的高质量、多源的预训练语料库。与DeepSeek 67B（我们之前的版本）（DeepSeek-AI |2024）中使用的语料库相比，该语料库具有更大的数据量，尤其是中文数据，并且具有更高的数据质量。我们首先在完整的预训练语料库上预训练 DeepSeek-V2。然后，我们收集 对话会话，这些会话涵盖数学、代码、写作、推理、安全等各个领域，以执行 DeepSeek-V2 聊天 （SFT） 的监督微调 （SFT）。最后，我们遵循 DeepSeekMath （Shao et al.， 2024） 采用组相对策略优化 （GRPO） 来进一步使模型与人类偏好保持一致并生成 DeepSeek-V2 聊天 （RL）。

We evaluate DeepSeek-V2 on a wide range of benchmarks in English and Chinese, and compare it with representative open-source models. Evaluation results show that even with only 21B activated parameters, DeepSeek-V2 still achieves top-tier performance among open-source models and becomes the strongest open-source MoE language model. Figure 1(a) highlights that, on MMLU, DeepSeek-V2 achieves top-ranking performance with only a small number of activated parameters. In addition, as shown in Figure 1(b), compared with DeepSeek 67B, DeepSeek-V2 saves of training costs, reduces the KV cache by , and boosts the maximum generation throughput to 5.76 times. We also evaluate DeepSeek-V2 Chat (SFT) and
我们在广泛的中英文基准测试中评估了 DeepSeek-V2，并将其与具有代表性的开源模型进行了比较。评估结果表明，即使仅激活了21B参数，DeepSeek-V2仍实现了开源模型中的顶级性能，成为最强的开源MoE语言模型。图 1（a） 突出显示，在 MMLU 上，DeepSeek-V2 仅使用少量激活参数即可实现顶级性能。此外，如图1（b）所示，与DeepSeek 67B相比，DeepSeek-V2节省 了训练成本，将KV缓存降低了 ，最大生成吞吐量提高到5.76倍。我们还评估了 DeepSeek-V2 聊天 （SFT） 和

Figure 1 | Illustration of the architecture of DeepSeek-V2. MLA ensures efficient inference by significantly reducing the KV cache for generation, and DeepSeekMoE enables training strong models at an economical cost through the sparse architecture.
图1 |DeepSeek-V2 架构图示。MLA 通过显著减少生成的 KV 缓存来确保高效推理，而 DeepSeekMoE 通过稀疏架构以经济的成本训练强大的模型。

DeepSeek-V2 Chat (RL) on open-ended benchmarks. Notably, DeepSeek-V2 Chat (RL) achieves 38.9 length-controlled win rate on AlpacaEval 2.0 (Dubois et al., 2024), 8.97 overall score on MT-Bench (Zheng et al. 2023), and 7.91 overall score on AlignBench (Liu et al., 2023). The English open-ended conversation evaluations demonstrate that DeepSeek-V2 Chat (RL) has toptier performance among open-source chat models. In addition, the evaluation on AlignBench indicates that in Chinese, DeepSeek-V2 Chat (RL) outperforms all of open-source models, and even beats most of closed-source models.
DeepSeek-V2 Chat （RL） 基于开放式基准测试。值得注意的是，DeepSeek-V2 Chat （RL） 在 AlpacaEval 2.0 （Dubois et al.， 2024） 上实现了 38.9 的长度控制胜率，在 MT-Bench 上获得了 8.97 的总分 （Zheng et al. 2023），在 AlignBench 上获得了 7.91 的总分 （Liu et al.， 2023）。英语开放式对话评估表明，DeepSeek-V2 Chat （RL） 在开源聊天模型中具有顶级性能。此外，AlignBench 的评估表明，在中文中，DeepSeek-V2 Chat （RL） 的表现优于所有开源模型，甚至优于大多数闭源模型。

In the rest of this paper, we first provide a detailed description of the model architecture of DeepSeek-V2 (Section 2). Subsequently, we introduce our pre-training endeavors, including the training data construction, hyper-parameter settings, infrastructures, long context extension, and the evaluation of model performance and efficiency (Section 3). Following this, we demonstrate our efforts in alignment, encompassing Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), the evaluation results, and other discussion (Section 4). Finally, we summarize the conclusion, deliberate on the current limitations of DeepSeek-V2, and outline our future work (Section 5).
在本文的其余部分，我们首先详细介绍了 DeepSeek-V2 的模型架构（第 2 节）。随后，我们介绍了我们的预训练工作，包括训练数据构建、超参数设置、基础设施、长上下文扩展以及模型性能和效率的评估（第 3 节）。在此之后，我们展示了我们在一致性方面的努力，包括监督微调 （SFT）、强化学习 （RL）、评估结果和其他讨论（第 4 节）。最后，我们总结了结论，讨论了 DeepSeek-V2 当前的局限性，并概述了我们未来的工作（第 5 节）。

By and large, DeepSeek-V2 is still in the Transformer architecture (Vaswani et al., 2017), where each Transformer block consists of an attention module and a Feed-Forward Network (FFN). However, for both the attention module and the FFN, we design and employ innovative architectures. For attention, we design MLA, which utilizes low-rank key-value joint compression to eliminate the bottleneck of inference-time key-value cache, thus supporting efficient inference. For FFNs, we adopt the DeepSeekMoE architecture (Dai et al., 2024), a high-performance MoE architecture that enables training strong models at an economical cost. An illustration of the architecture of DeepSeek-V2 is presented in Figure 1, and we will introduce the details of MLA and DeepSeekMoE in this section. For other tiny details (e.g., layer normalization and the activation function in FFNs), unless specifically stated, DeepSeek-V2 follows the settings of DeepSeek 67B ( .
总的来说，DeepSeek-V2 仍处于 Transformer 架构中（Vaswani 等人，2017 年），其中每个 Transformer 模块由一个注意力模块和一个前馈网络 （FFN） 组成。然而，对于注意力模块和FFN，我们设计和采用了创新的架构。为了引起注意，我们设计了MLA，它利用低秩键值联合压缩来消除推理-时间键值缓存的瓶颈，从而支持高效推理。对于 FFN，我们采用 DeepSeekMoE 架构（Dai et al.， 2024），这是一种高性能的 MoE 架构，能够以经济的成本训练强大的模型。图 1 展示了 DeepSeek-V2 的架构，我们将在本节中详细介绍 MLA 和 DeepSeekMoE。对于其他微小的细节（例如，层归一化和 FFN 中的激活函数），除非特别说明，否则 DeepSeek-V2 遵循 DeepSeek 67B （ .

Conventional Transformer models usually adopts Multi-Head Attention (MHA) (Vaswani et al. 2017), but during generation, its heavy Key-Value (KV) cache will become the bottleneck that limit the inference efficiency. In order to reduce the KV cache, Multi-Query Attention (MQA) (Shazeer, 2019) and Grouped-Query Attention (GQA) (Ainslie et al., 2023) are proposed. They require a smaller magnitude of cache, but their performance does not match MHA (we provide the ablation of MHA, GQA and MQA in Appendix C.1).
传统的 Transformer 模型通常采用多头注意力 （MHA） （Vaswani et al. 2017），但在生成过程中，其繁重的键值 （KV） 缓存将成为限制推理效率的瓶颈。为了减少KV缓存，提出了多查询注意力（MQA）（Shazeer，2019）和分组查询注意力（GQA）（Ainslie等人，2023）。它们需要较小量级的 缓存，但它们的性能与 MHA 不匹配（我们在附录 C.1 中提供了 MHA、GQA 和 MQA 的消融）。

For DeepSeek-V2, we design an innovative attention mechanism called Multi-head Latent Attention (MLA). Equipped with low-rank key-value joint compression, MLA achieves better performance than MHA, but requires a significantly smaller amount of cache. We introduce its architecture in the following, and also provide a comparison between MLA and MHA in Appendix
对于 DeepSeek-V2，我们设计了一种创新的注意力机制，称为多头潜在注意力 （MLA）。MLA 配备低秩键值联合压缩，性能优于 MHA，但需要的 缓存量要小得多。我们将在下面介绍其架构，并在附录 中提供 MLA 和 MHA 之间的比较

We first introduce the standard MHA mechanism as background. Let be the embedding dimension, be the number of attention heads, be the dimension per head, and be the attention input of the -th token at an attention layer. Standard MHA first produces through three matrices , respectively:
我们首先介绍标准的MHA机制作为背景。设 是嵌入维度， 是注意力头的数量， 是每个头的维度， 并且是注意力层上第 -th 标记的注意力输入。标准MHA首先通过三个矩阵 分别产生 ：

Then, will be sliced into heads for the multi-head attention computation:
然后， 将被切成 头部进行多头注意力计算：

Figure 2 | Simplified illustration of Multi-Head Attention (MHA), Grouped-Query Attention (GQA), Multi-Query Attention (MQA), and Multi-head Latent Attention (MLA). Through jointly compressing the keys and values into a latent vector, MLA significantly reduces the KV cache during inference.
图2 |多头注意力 （MHA）、分组查询注意力 （GQA）、多查询注意力 （MQA） 和多头潜在注意力 （MLA） 的简化图示。通过将键和值联合压缩为潜在向量，MLA显著降低了推理过程中的KV缓存。

where denote the query, key, and value of the -th attention head, respectively; denotes the output projection matrix. During inference, all keys and values need to be cached to accelerate inference, so MHA needs to cache elements for each token. In model deployment, this heavy KV cache is a large bottleneck that limits the maximum batch size and sequence length.
其中 分别表示 第 -th 个注意力头的查询、键和值; 表示输出投影矩阵。在推理过程中，需要缓存所有键和值以加速推理，因此 MHA 需要缓存每个令牌 的元素。在模型部署中，这种繁重的 KV 缓存是一个很大的瓶颈，它限制了最大批处理大小和序列长度。

The core of MLA is the low-rank joint compression for keys and values to reduce cache:
MLA 的核心是对键和值进行低秩联合压缩，以减少 缓存：

where is the compressed latent vector for keys and values; denotes the compression dimension; is the down-projection matrix; and are the up-projection matrices for keys and values, respectively. During inference, MLA only needs to cache , so its cache has only elements, where denotes the number of layers. In addition, during inference, since can be absorbed into , and can be absorbed into , we even do not need to compute keys and values out for attention. Figure 2 intuitively illustrates how the joint compression in MLA reduces the cache.
其中 是键和值的压缩潜在向量; 表示 压缩尺寸; 是向下投影矩阵;和 分别是键和值的上投影矩阵。在推理过程中，MLA 只需要缓存 ，因此其 缓存只有 元素，其中 表示层数。此外，在推理过程中，由于 可以被吸收到 和 可以吸收到 ，我们甚至不需要计算出键和值来引起注意。图 2 直观地说明了 MLA 中 的联合压缩如何减少 缓存。

Moreover, in order to reduce the activation memory during training, we also perform low-rank compression for the queries, even if it cannot reduce the cache:
此外，为了减少训练过程中的激活内存，我们还对查询进行了低秩压缩，即使它不能减少 缓存：

where is the compressed latent vector for queries; denotes the query compression dimension; and are the down-projection and upprojection matrices for queries, respectively.
其中 是查询的压缩潜在向量; 表示查询压缩维度;和 分别是查询的下投影和上投影矩阵。

Following DeepSeek 67B ( ) we intend to use the Rotary Position Embedding (RoPE) (Su et al., 2024) for DeepSeek-V2. However, RoPE is incompatible with low-rank
在 DeepSeek 67B （ ） 之后，我们打算将旋转位置嵌入 （RoPE） （Su et al.， 2024） 用于 DeepSeek-V2。但是，RoPE 与低秩不兼容

Table 1 | Comparison of the KV cache per token among different attention mechanisms. denotes the number of attention heads, denotes the dimension per attention head, denotes the number of layers, denotes the number of groups in GQA, and and denote the KV compression dimension and the per-head dimension of the decoupled queries and key in MLA, respectively. The amount of cache is measured by the number of elements, regardless of the storage precision. For DeepSeek-V2, is set to and is set to . So, its KV cache is equal to GQA with only 2.25 groups, but its performance is stronger than MHA.
表1 |不同注意力机制下每个令牌的 KV 缓存比较。 分别表示注意力头数、 每个注意力头的维度、 层数、 GQA 中的组数、MLA中解耦查询和键的KV压缩维度和每头维度。 缓存量由元素数来衡量，而与存储精度无关。对于 DeepSeek-V2， 设置为 和 设置为 。因此，它的 KV 缓存等于 GQA，只有 2.25 个组，但其性能比 MHA 强。

KV compression. To be specific, RoPE is position-sensitive for both keys and queries. If we apply RoPE for the keys in Equation 10 will be coupled with a position-sensitive RoPE matrix. In this way, cannot be absorbed into any more during inference, since a RoPE matrix related to the currently generating token will lie between and and matrix multiplication does not obey a commutative law. As a result, we must recompute the keys for all the prefix tokens during inference, which will significantly hinder the inference efficiency.
KV压缩。具体来说，RoPE 对键和查询都是位置敏感的。如果我们对公式 10 中的键 应用 RoPE，则将与位置敏感的 RoPE 矩阵耦合。这样， 在推理过程中就不能再被吸收了 ，因为与当前生成的令牌相关的 RoPE 矩阵将位于 和 之间 ，并且矩阵乘法不遵守交换定律。因此，在推理过程中，我们必须重新计算所有前缀标记的键，这将大大阻碍推理效率。

As a solution, we propose the decoupled RoPE strategy that uses additional multi-head queries and a shared key to carry RoPE, where denotes the per-head dimension of the decoupled queries and key. Equipped with the decoupled RoPE strategy, MLA performs the following computation:
作为解决方案，我们提出了解耦的 RoPE 策略，该策略使用额外的多头查询 和共享密钥 来携带 RoPE，其中 表示解耦查询和键的每头维度。配备解耦 RoPE 策略，MLA 执行以下计算：

where and are matrices to produce the decouples queries and key, respectively; denotes the operation that applies RoPE matrices; and denotes the concatenation operation. During inference, the decoupled key should also be cached. Therefore, DeepSeek-V2 requires a total cache containing elements.
其中 和 是矩阵，分别产生解耦查询和 key; 表示应用 RoPE 矩阵的操作;并表示 串联运算。在推理过程中，还应该缓存解耦的密钥。因此，DeepSeek-V2 需要包含 元素的总 缓存。

In order to demonstrate the complete computation process of MLA, we also organize and provide its full formulas in Appendix B.
为了演示MLA的完整计算过程，我们还在附录B中整理并提供了其完整的公式。

We demonstrate a comparison of the KV cache per token among different attention mechanisms in Table 1 MLA requires only a small amount of KV cache, equal to GQA with only 2.25 groups, but can achieve stronger performance than MHA.
我们在表 1 中展示了不同注意力机制之间每个令牌的 KV 缓存的比较，MLA 只需要少量的 KV 缓存，相当于只有 2.25 个组的 GQA，但可以实现比 MHA 更强的性能。

For FFNs, we employ the DeepSeekMoE architecture (Dai et al., 2024). DeepSeekMoE has two key ideas: segmenting experts into finer granularity for higher expert specialization and more accurate knowledge acquisition, and isolating some shared experts for mitigating knowledge redundancy among routed experts. With the same number of activated and total expert parameters, DeepSeekMoE can outperform conventional MoE architectures like GShard (Lepikhin et al., 2021) by a large margin.
对于 FFN，我们采用 DeepSeekMoE 架构（Dai 等人，2024 年）。DeepSeekMoE 有两个关键思想：将专家细分为更精细的粒度，以实现更高的专家专业化和更准确的知识获取，以及隔离一些共享专家以减少路由专家之间的知识冗余。在相同数量的激活和总专家参数下，DeepSeekMoE 的性能可以大大优于 GShard 等传统 MoE 架构（Lepikhin 等人，2021 年）。

Let be the FFN input of the -th token, we compute the FFN output as follows:
假设 是 第 -th 个标记的 FFN 输入，我们按如下方式计算 FFN 输出 ：

where and denote the numbers of shared experts and routed experts, respectively; and denote the -th shared expert and the -th routed expert, respectively; denotes the number of activated routed experts; is the gate value for the -th expert; is the tokento-expert affinity; is the centroid of the -th routed expert in this layer; and denotes the set comprising highest scores among the affinity scores calculated for the -th token and all routed experts.
其中 和 分别表示共享专家和路由专家的数量; 分别 表示 -th 共享专家和 -th 路由专家; 表示已激活的路由专家的数量; 是 第 -th 个 expert 的门值; 是令牌与专家的亲和力; 是该层中 第 -th 路由专家的质心;并表示 包含针对 第 -th 个标记和所有路由专家计算的亲和力分数 中最高分数的集合。

We design a device-limited routing mechanism to bound MoE-related communication costs. When expert parallelism is employed, the routed experts will be distributed across multiple devices. For each token, its MoE-related communication frequency is proportional to the number of devices covered by its target experts. Due to the fine-grained expert segmentation in DeepSeekMoE, the number of activated experts can be large, so the MoE-related communication will be more costly if we apply expert parallelism.
我们设计了一种设备受限的路由机制来绑定与 MoE 相关的通信成本。当采用专家并行时，路由的专家将分布在多个设备上。对于每个令牌，其与 MoE 相关的通信频率与其目标专家覆盖的设备数量成正比。由于 DeepSeekMoE 中细粒度的专家分段，激活的专家数量可能很大，因此如果我们应用专家并行，与 MoE 相关的通信成本会更高。

For DeepSeek-V2, beyond the naive top-K selection of routed experts, we additionally ensure that the target experts of each token will be distributed on at most devices. To be specific, for each token, we first select devices that have experts with the highest affinity scores in them. Then, we perform top-K selection among experts on these devices. In practice, we find that when , the device-limited routing can achieve a good performance roughly aligned with the unrestricted top- routing.
对于 DeepSeek-V2，除了朴素的 top-K 路由专家选择之外，我们还确保每个代币的目标专家将分布在大多数 设备上。具体来说，对于每个令牌，我们首先选择 具有具有最高亲和力分数的专家的设备。然后，我们在这些 设备的专家中执行 top-K 选择。在实践中，我们发现，当 时，设备受限的路由可以实现与不受限制的顶部 路由大致一致的良好性能。

We take the load balance into consideration for automatically learned routing strategies. Firstly, unbalanced load will raise the risk of routing collapse (Shazeer et al., 2017), preventing some experts being fully trained and utilized. Secondly, when expert parallelism is employed, unbalanced load will diminish computation efficiency. During the training of DeepSeek-V2, we design three kinds of auxiliary losses, for controlling expert-level load balance ( ), device-level load balance , and communication balance , respectively.
对于自动学习的路由策略，我们会考虑负载均衡。首先，不平衡的负载会增加路由崩溃的风险（Shazeer et al.， 2017），使一些专家无法得到充分的培训和利用。其次，当采用专家并行时，不平衡负载会降低计算效率。在DeepSeek-V2的训练过程中，我们设计了三种辅助损耗，分别用于控制专家级负载均衡（ ）、设备级负载均衡 和通信均衡 。

Expert-Level Balance Loss. We use an expert-level balance loss (Fedus et al. 2021 ; Lepikhin et al. 2021) to mitigate the risk of routing collapse:
专家级余额损失。我们使用专家级余额损失（Fedus 等人，2021 年;Lepikhin 等人，2021 年）来降低路由崩溃的风险：

where is a hyper-parameter called expert-level balance factor; denotes the indicator function; and denotes the number of tokens in a sequence.
其中 是一个称为专家级平衡因子的超参数; 表示指标功能;并表示 序列中的标记数。

Device-Level Balance Loss. In addition to the expert-level balance loss, we additionally design a device-level balance loss to ensure balanced computation across different devices. In the training process of DeepSeek-V2, we partition all routed experts into groups , and deploy each group on a single device. The device-level balance loss is computed as follows:
设备级余额损失。除了专家级的平衡损失外，我们还设计了设备级的平衡损失，以确保不同设备之间的平衡计算。在 DeepSeek-V2 的训练过程中，我们将所有路由专家划分为 组 ，并将每个组部署在单个设备上。设备级余额损失的计算方法如下：

where is a hyper-parameter called device-level balance factor.
其中 是称为设备级平衡因子的超参数。

Communication Balance Loss. Finally, we introduce a communication balance loss to ensure that the communication of each device is balanced. Although the device-limited routing mechanism guarantees that the sending communication of each device is bounded, if a certain device receives more tokens than other devices, the practical communication efficiency will also be affected. In order to mitigate this issue, we design a communication balance loss as follows:
通信余额损失。最后，我们引入通信平衡损失，以确保每个设备的通信是平衡的。虽然设备限制的路由机制保证了每个设备的发送通信是有界的，但如果某个设备接收到的令牌比其他设备多，实际通信效率也会受到影响。为了缓解这个问题，我们设计了一个通信余额损失，如下所示：

where is a hyper-parameter called communication balance factor. The device-limited routing mechanism operates on the principle of ensuring that each device transmits at most hidden states to other devices. Simultaneously, the communication balance loss is employed to encourage each device to receive around hidden states from other devices. The communication balance loss guarantees a balanced exchange of information among devices, promoting efficient communications.
其中 是称为通信平衡因子的超参数。设备限制路由机制的运行原理是确保每个设备最多 以隐藏状态传输到其他设备。同时，通信平衡损失用于鼓励每个设备接收来自其他设备的隐藏 状态。通信平衡损失保证了设备之间的信息平衡交换，促进了高效的通信。

While balance losses aim to encourage a balanced load, it is important to acknowledge that they cannot guarantee a strict load balance. In order to further mitigate the computation wastage caused by unbalanced load, we introduce a device-level token-dropping strategy during training. This approach first computes the average computational budget for each device, which means that the capacity factor for each device is equivalent to 1.0. Then, inspired by Riquelme et al. (2021), we drop tokens with the lowest affinity scores on each device until reaching the computational budget. In addition, we ensure that the tokens belonging to approximately of the training sequences will never be dropped. In this way, we can flexibly decide whether to drop tokens during inference according to the efficiency requirements, and always ensure consistency between training and inference.
虽然平衡损失旨在鼓励平衡负载，但重要的是要承认它们不能保证严格的负载平衡。为了进一步缓解负载不均衡导致的计算浪费，我们在训练过程中引入了设备级令牌丢弃策略。此方法首先计算每个设备的平均计算预算，这意味着每个设备的容量因子等于 1.0。然后，受 Riquelme 等人 （2021） 的启发，我们在每台设备上丢弃亲和力分数最低的代币，直到达到计算预算。此外，我们确保属于大约 训练序列的标记永远不会被丢弃。这样，我们可以根据效率要求灵活地决定在推理过程中是否丢弃令牌，并始终保证训练和推理的一致性。

While maintaining the same data processing stages as for DeepSeek 67B (DeepSeek-AI, 2024), we extend the amount of data and elevate the data quality. In order to enlarge our pre-training corpus, we delve deeper into the potential of the internet data and optimize our cleaning processes, thus recovering a large amount of mistakenly deleted data. Moreover, we incorporate more Chinese data, aiming to better leverage the corpus available on the Chinese internet. In addition to the amount of data, we also focus on the data quality. We enrich our pre-training corpus with high-quality data from various sources, and meanwhile improve the quality-based filtering algorithm. The improved algorithm ensures that a large amount of non-beneficial data will be removed, while the valuable data will be mostly retained. In addition, we filter out the contentious content from our pre-training corpus to mitigate the data bias introduced from specific regional cultures. A detailed discussion about the influence of this filtering strategy is presented in Appendix
在保持与 DeepSeek 67B（DeepSeek-AI，2024）相同的数据处理阶段的同时，我们扩展了数据量并提高了数据质量。为了扩大我们的预训练语料库，我们深入研究了互联网数据的潜力并优化了我们的清理过程，从而恢复了大量错误删除的数据。此外，我们纳入了更多的中文数据，旨在更好地利用中国互联网上的语料库。除了数据量，我们还关注数据质量。我们用来自各种来源的高质量数据丰富了我们的预训练语料库，同时改进了基于质量的过滤算法。改进后的算法确保了大量无益数据将被删除，而有价值的数据将大部分被保留。此外，我们从预训练语料库中过滤掉有争议的内容，以减轻从特定区域文化引入的数据偏差。附录 中详细介绍了这种过滤策略的影响

We adopt the same tokenizer as used in DeepSeek 67B, which is built based on the Byte-level Byte-Pair Encoding (BBPE) algorithm and has a vocabulary size of 100K. Our tokenized pretraining corpus contains tokens, where Chinese tokens are approximately more than English ones.
我们采用与 DeepSeek 67B 相同的分词器，该分词器基于字节级字节对编码 （BBPE） 算法构建，词汇量为 100K。我们的代币化预训练语料库包含 代币，其中中文代币大约 多于英文代币。

Model Hyper-Parameters. We set the number of Transformer layers to 60 and the hidden dimension to 5120 . All learnable parameters are randomly initialized with a standard deviation of 0.006. In MLA, we set the number of attention heads to 128 and the per-head dimension to 128 . The compression dimension is set to 512 , and the query compression dimension is set to 1536. For the decoupled queries and key, we set the per-head dimension to 64 . Following Dai et al. (2024), we substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 2 shared experts and 160 routed experts, where the intermediate hidden dimension of each expert is 1536. Among the routed experts, 6 experts will be activated for each token. In addition, the low-rank compression and fine-grained expert segmentation will impact the output scale of a layer. Therefore, in practice, we employ additional RMS Norm layers after the compressed latent vectors, and multiply additional scaling factors at the width bottlenecks (i.e., the compressed latent vectors and the intermediate hidden states of routed
对超参数进行建模。我们将 Transformer 层数设置为 60，将隐藏维度设置为 5120。所有可学习的参数都是随机初始化的，标准差为 0.006。在 MLA 中，我们将注意力头 的数量设置为 128，将每个头的维度 设置为 128。 压缩维度 设置为 512，查询压缩维度 设置为 1536。对于解耦的查询和键，我们将每头维度 设置为 64 。根据 Dai 等人 （2024），我们将除第一层以外的所有 FFN 替换为 MoE 层。每个 MoE 层由 2 个共享专家和 160 个路由专家组成，其中每个专家的中间隐藏维度为 1536。在路由的专家中，每个代币将激活 6 名专家。此外，低秩压缩和细粒度专家分割会影响层的输出规模。因此，在实践中，我们在压缩的潜在向量之后使用额外的RMS范数层，并在宽度瓶颈处乘以额外的比例因子（即压缩的潜在向量和路由的中间隐藏状态）
experts) to ensure stable training. Under this configuration, DeepSeek-V2 comprises 236B total parameters, of which 21B are activated for each token.
专家）以确保稳定的培训。在这种配置下，DeepSeek-V2 总共包含 236B 个参数，其中 21B 为每个令牌激活。

Training Hyper-Parameters. We employ the AdamW optimizer (Loshchilov and Hutter, 2017) with hyper-parameters set to , and weight_decay . The learning rate is scheduled using a warmup-and-step-decay strategy (DeepSeek-AI, 2024). Initially, the learning rate linearly increases from 0 to the maximum value during the first steps. Subsequently, the learning rate is multiplied by 0.316 after training about of tokens, and again by 0.316 after training about of tokens. The maximum learning rate is set to , and the gradient clipping norm is set to 1.0 . We also use a batch size scheduling strategy, where the batch size is gradually increased from 2304 to 9216 in the training of the first 225B tokens, and then keeps 9216 in the remaining training. We set the maximum sequence length to , and train DeepSeek-V2 on 8.1T tokens. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, the routed experts will be uniformly deployed on 8 devices . As for the device-limited routing, each token will be sent to at most 3 devices . As for balance losses, we set to to 0.05 , and to 0.02 . We employ the token-dropping strategy during training for acceleration, but do not drop any tokens for evaluation.
训练超参数。我们使用 AdamW 优化器（Loshchilov 和 Hutter，2017 年），超参数设置为 ，并weight_decay 。学习率是使用预热和步进衰减策略安排的（DeepSeek-AI，2024）。最初，在第一 步中，学习率从 0 线性增加到最大值。随后，在训练有关 令牌后，学习率乘以 0.316，在训练有关 令牌后再次乘以 0.316。最大学习率设置为 ，梯度削波范数设置为 1.0。我们还使用批量大小调度策略，在前 225B 代币的训练中，批大小从 2304 逐渐增加到 9216，然后在剩余的训练中保留 9216。我们将最大序列长度设置为 ，并在 8.1T 代币上训练 DeepSeek-V2。我们利用流水线并行性将模型的不同层部署在不同的设备上，对于每一层，路由的专家将统一部署在 8 台设备 上。至于设备限制的路由，每个令牌最多会发送到 3 台设备 。至于余额损失，我们设置为 0.05 和 0.02 。我们在加速训练期间采用令牌丢弃策略，但不会丢弃任何令牌进行评估。

DeepSeek-V2 is trained based on the HAI-LLM framework (High-flyer 2023), an efficient and light-weight training framework developed internally by our engineers. It employs a 16-way zero-bubble pipeline parallelism (Qi et al., 2023), an 8-way expert parallelism (Lepikhin et al. 2021), and ZeRO-1 data parallelism (Rajbhandari et al., 2020). Given that DeepSeek-V2 has relatively few activated parameters, and a portion of the operators are recomputed to save activation memory, it can be trained without the necessity of tensor parallelism, thereby decreasing the communication overhead. Moreover, in order to further improve the training efficiency, we overlap the computation of shared experts with the expert parallel all-to-all communication. We also customize faster CUDA kernels for communications, routing algorithms, and fused linear computations across different experts. In addition, MLA is also optimized based on an improved version of FlashAttention-2 (Dao, 2023).
DeepSeek-V2 基于 HAILLM 框架 （High-flyer 2023） 进行训练，这是一个由我们的工程师内部开发的高效、轻量级的训练框架。它采用 16 路零气泡管道并行性（Qi 等人，2023 年）、8 路专家并行性（Lepikhin 等人，2021 年）和 ZeRO-1 数据并行性（Rajbhandari 等人，2020 年）。鉴于 DeepSeek-V2 的激活参数相对较少，并且重新计算了一部分算子以节省激活内存，因此无需张量并行即可对其进行训练，从而降低了通信开销。此外，为了进一步提高训练效率，我们将共享专家的计算与专家并行多对多通信重叠。我们还为不同专家的通信、路由算法和融合线性计算定制更快的 CUDA 内核。此外，MLA 还基于 FlashAttention-2 的改进版本进行了优化（Dao，2023）。

We conduct all experiments on a cluster equipped with NVIDIA H800 GPUs. Each node in the H800 cluster contains 8 GPUs connected using NVLink and NVSwitch within nodes. Across nodes, InfiniBand interconnects are utilized to facilitate communications.
我们在配备 NVIDIA H800 GPU 的集群上进行所有实验。H800 集群中的每个节点包含 8 个 GPU，在节点内使用 NVLink 和 NVSwitch 连接。跨节点，利用 InfiniBand 互连来促进通信。

After the initial pre-training of DeepSeek-V2, we employ YaRN (Peng et al., 2023) to extend the default context window length from to . YaRN was specifically applied to the decoupled shared key as it is responsible for carrying RoPE (Su et al. 2024). For YaRN, we set the scale to to to 32 , and the target maximum context length to . Under these settings, we can expect the model to respond well for a context length of . Slightly diverging from original YaRN, due to our distinct attention mechanism, we adjust the length scaling factor to modulate the attention entropy. The factor is computed as , aiming at minimizing the perplexity.
在对 DeepSeek-V2 进行初始预训练后，我们使用 YaRN（Peng 等人，2023 年）将默认上下文窗口长度从 扩展到 。YaRN 专门应用于解耦共享密钥 ，因为它负责携带 RoPE（Su 等人，2024 年）。对于 YaRN，我们将比例 设置为 32 ，并将目标最大上下文长度设置为 。在这些设置下，我们可以预期模型对上下文长度 的 响应良好。与原始 YaRN 略有不同，由于我们独特的注意力机制，我们调整了长度比例因子来调节注意力熵。该因子 计算为 ，旨在将困惑度降至最低。

We additionally train the model for 1000 steps, with a sequence length of and a batch size of 576 sequences. Although the training is conducted solely at the sequence length of ,
此外，我们还训练了 1000 个步骤的模型，序列长度为 576 个序列，批处理大小为 576 个序列。虽然训练仅在 的序列长度上 进行

Figure 3 | Evaluation results on the "Needle In A Haystack" (NIAH) tests. DeepSeek-V2 performs well across all context window lengths up to .
图3 |“大海捞针”（NIAH）测试的评估结果。DeepSeek-V2 在所有上下文窗口长度上都表现良好，最高 可达 。

the model still demonstrates robust performance when being evaluated at a context length of 128K. As shown in Figure 3, the results on the "Needle In A Haystack" (NIAH) tests indicate that DeepSeek-V2 performs well across all context window lengths up to .
在 128K 的上下文长度下进行评估时，该模型仍表现出强大的性能。如图 3 所示，“大海捞针”（NIAH） 测试的结果表明，DeepSeek-V2 在所有上下文窗口长度上都表现良好，最高 可达 。

DeepSeek-V2 is pretrained on a bilingual corpus, so we evaluate it on a series of benchmarks in English and Chinese. Our evaluation is based on our internal evaluation framework integrated in our HAI-LLM framework. Included benchmarks are categorized and listed as follows, where underlined benchmarks are in Chinese:
DeepSeek-V2 是在双语语料库上预先训练的，因此我们在一系列英文和中文基准测试中对其进行了评估。我们的评估基于集成在HAILLM框架中的内部评估框架。所包含的基准标准分类如下，其中带下划线的基准为中文基准：

Multi-subject multiple-choice datasets include MMLU (Hendrycks et al., 2020), C-Eval (Huang et al. 2023), and CMMLU (Li et al., 2023).
多主题多项选择数据集包括 MMLU（Hendrycks 等人，2020 年）、C-Eval （Huang 等人，2023 年）和 CMMLU（Li 等人，2023 年）。

Language understanding and reasoning datasets include HellaSwag (Zellers et al. 2019), PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), and BigBench Hard (BBH) (Suzgun et al., 2022).
语言理解和推理数据集包括 HellaSwag （Zellers et al. 2019）、PIQA （Bisk et al.， 2020）、ARC （Clark et al.， 2018） 和 BigBench Hard （BBH） （Suzgun et al.， 2022）。

Closed-book question answering datasets include TriviaQA (Joshi et al. 2017) and NaturalQuestions (Kwiatkowski et al., 2019).
闭卷问答数据集包括 TriviaQA （Joshi et al. 2017） 和 NaturalQuestions （Kwiatkowski et al.， 2019）。

Reading comprehension datasets include RACE Lai et al. (2017), DROP (Dua et al. 2019), C3 (Sun et al. 2019), and CMRC (Cui et al. 2019).
阅读理解数据集包括 RACE Lai et al. （2017）、DROP （Dua et al. 2019）、C3 （Sun et al. 2019） 和 CMRC （Cui et al. 2019）。

Reference disambiguation datasets include WinoGrande Sakaguchi et al.(2019) and CLUEWSC (Xu et al. 2020 ).
参考消歧数据集包括 WinoGrande Sakaguchi et al.（2019） 和 CLUEWSC （Xu et al. 2020）。

Language modeling datasets include Pile (Gao et al., 2020).
语言建模数据集包括 Pile （Gao et al.， 2020）。

Chinese understanding and culture datasets include (Zheng et al., 2019) and CCPM (Li et al. 2021 ).
中国的理解和文化数据集包括 （Zheng et al.， 2019） 和 CCPM （Li et al. 2021）。

Math datasets include GSM8K (Cobbe et al. 2021), MATH (Hendrycks et al. , 2021), and CMath (Wei et al. 2023 ).
数学数据集包括 GSM8K（Cobbe 等人，2021 年）、数学（Hendrycks 等人，2021 年）和 CMath（Wei 等人，2023 年）。

Code datasets include HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and CRUXEval (Gu et al., 2024).
代码数据集包括 HumanEval （Chen et al.， 2021）、MBPP （Austin et al.， 2021） 和 CRUXEval （Gu et al.， 2024）。

Standardized exams include AGIEval (Zhong et al., 2023). Note that AGIEval includes both English and Chinese subsets.
标准化考试包括 AGIEval（Zhong 等人，2023 年）。请注意，AGIEval 包括英文和中文子集。

Following our previous work (DeepSeek-AI, 2024), we adopt perplexity-based evaluation for datasets including HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, CHID, C-Eval, CMMLU, C3, and CCPM, and adopt generationbased evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, CRUXEval, BBH, AGIEval, CLUEWSC, CMRC, and CMath. In addition, we perform languagemodeling-based evaluation for Pile-test and use Bits-Per-Byte (BPB) as the metric to guarantee fair comparison among models with different tokenizers.
根据我们之前的工作（DeepSeek-AI，2024），我们对包括 HellaSwag、PIQA、WinoGrande、RACE-Middle、RACE-High、MMLU、ARC-Easy、ARC-Challenge、CHID、C-Eval、CMMLU、C3 和 CCPM 在内的数据集采用基于困惑的评估，并对 TriviaQA、NaturalQuestions、DROP、MATH、GSM8K、HumanEval、MBPP、CRUXEval、BBH、AGIEval、CLUEWSC、CMRC 和 CMath 等数据集采用基于世代的评估。此外，我们还对 Pile-test 进行基于语言建模的评估，并使用每字节比特 （BPB） 作为指标，以保证不同分词器模型之间的公平比较。

For an intuitive overview of these benchmarks, we additionally provide our evaluation formats for each benchmark in Appendix F.
为了直观地了解这些基准，我们还在附录 F 中提供了每个基准的评估格式。

In Table 2, we compare DeepSeek-V2 with several representative open-source models, including DeepSeek 67B (DeepSeek-AI, 2024) (our previous release), Qwen1.5 72B (Bai et al., 2023), LLaMA3 70B (AI@Metal, 2024), and Mixtral 8x22B (Mistral, 2024). We evaluate all these models with our internal evaluation framework, and ensure that they share the same evaluation setting. Overall, with only 21B activated parameters, DeepSeek-V2 significantly outperforms DeepSeek 67B on almost all benchmarks, and achieves top-tier performance among open-source models.
在表 2 中，我们将 DeepSeek-V2 与几个具有代表性的开源模型进行了比较，包括 DeepSeek 67B （DeepSeek-AI， 2024） （我们之前的版本）、Qwen1.5 72B （Bai et al.， 2023）、LLaMA3 70B （AI@Metal， 2024） 和 Mixtral 8x22B （Mistral， 2024）。我们使用内部评估框架评估所有这些模型，并确保它们共享相同的评估设置。总体而言，仅激活 21B 参数的 DeepSeek-V2 在几乎所有基准测试中都明显优于 DeepSeek 67B，并在开源模型中实现了顶级性能。

Further, we elaborately compare DeepSeek-V2 with its open-source counterparts one by one. (1) Compared with Qwen1.5 72B, another model that supports both Chinese and English, DeepSeek-V2 demonstrates overwhelming advantages on the majority of English, code, and math benchmarks. As for Chinese benchmarks, Qwen1.5 72B shows better performance on multi-subject multiple-choice tasks while DeepSeek-V2 is comparable or better on others. Note that for the CHID benchmark, the tokenizer of Qwen1.5 72B will encounter errors in our evaluation framework, so we leave the CHID score blank for Qwen1.5 72B. (2) Compared with Mixtral 8x22B, DeepSeek-V2 achieves comparable or better English performance, except for TriviaQA, NaturalQuestions, and HellaSwag, which are closely related to English commonsense knowledge. Notably, DeepSeek-V2 outperforms Mixtral 8x22B on MMLU. On code and math benchmarks, DeepSeek-V2 demonstrates comparable performance with Mixtral B. Since Mixtral 8x22B is not specifically trained on Chinese data, its Chinese capability lags far behind DeepSeek-V2. (3) Compared with LLaMA3 70B, DeepSeek-V2 is trained on fewer than a quarter of English tokens. Therefore, we acknowledge that DeepSeek-V2 still has a slight gap in basic English capabilities with LLaMA3 70B. However, even with much fewer training tokens and activated parameters, DeepSeek-V2 still demonstrates comparable code and math capability with LLaMA3 70B. Also, as a bilingual language model, DeepSeek-V2 outperforms LLaMA3 70B overwhelmingly on Chinese benchmarks.
此外，我们详细地将 DeepSeek-V2 与其开源对应物进行了逐一比较。（1）与另一款同时支持中英文的Qwen1.5 72B相比，DeepSeek-V2在大多数英语、代码和数学基准测试中表现出压倒性的优势。在中国的基准测试中，Qwen1.5 72B在多主题多项选择任务上表现出更好的表现，而DeepSeek-V2在其他基准测试中的表现相当或更好。需要注意的是，对于CHID基准测试，Qwen1.5 72B的分词器在我们的评估框架中会遇到错误，因此我们将Qwen1.5 72B的CHID分数留空。 （2）与Mixtral 8x22B相比，DeepSeek-V2的英语性能相当或更好，但TriviaQA、NaturalQuestions和HellaSwag除外，它们与英语常识知识密切相关。值得注意的是，DeepSeek-V2 在 MMLU 上的性能优于 Mixtral 8x22B。在代码和数学基准测试中，DeepSeek-V2 的性能与 Mixtral B 相当。由于 Mixtral 8x22B 没有专门针对中国数据进行训练，因此其中文能力远远落后于 DeepSeek-V2。（3）与LLaMA3 70B相比，DeepSeek-V2的训练使用不到四分之一的英文代币。因此，我们承认 DeepSeek-V2 在基本英语功能方面与 LLaMA3 70B 相比仍存在轻微差距。然而，即使训练令牌和激活参数少得多，DeepSeek-V2 仍然展示了与 LLaMA3 70B 相当的代码和数学能力。此外，作为双语语言模型，DeepSeek-V2 在中国基准测试中的表现压倒性地优于 LLaMA3 70B。

Finally, it is worth mentioning that certain prior studies (Hu et al., 2024) incorporate SFT data during the pre-training stage, whereas DeepSeek-V2 has never been exposed to SFT data during pre-training.
最后，值得一提的是，某些先前的研究（胡等人，2024）在预训练阶段纳入了SFT数据，而DeepSeek-V2在预训练期间从未接触过SFT数据。