DeepSeek-V3 Technical Report
DeepSeek-V3 技术报告

For attention, DeepSeek-V3 adopts the MLA architecture. Let $d$ denote the embedding dimension, $n_{h}$ denote the number of attention heads, $d_{h}$ denote the dimension per head, and $\mathbf{h}_{t}\in\mathbb{R}^{d}$ denote the attention input for the $t$-th token at a given attention layer. The core of MLA is the low-rank joint compression for attention keys and values to reduce Key-Value (KV) cache during inference:
DeepSeek-V3 采用 MLA 架构。用 $d$ 表示嵌入维度， $n_{h}$ 表示注意力头数， $d_{h}$ 表示每个头的维度， $\mathbf{h}_{t}\in\mathbb{R}^{d}$ 表示给定注意力层中第 $t$ 个标记的注意力输入。MLA 的核心是对注意力键和值进行低秩联合压缩，以减少推理过程中的键值（KV）缓存：

where $\mathbf{c}_{t}^{KV}\in\mathbb{R}^{d_{c}}$ is the compressed latent vector for keys and values; $d_{c}(\ll d_{h}n_{h})$ indicates the KV compression dimension; $W^{DKV}\in\mathbb{R}^{d_{c}\times d}$ denotes the down-projection matrix; $W^{UK},W^{UV}\in\mathbb{R}^{d_{h}n_{h}\times d_{c}}$ are the up-projection matrices for keys and values, respectively; $W^{KR}\in\mathbb{R}^{d_{h}^{R}\times d}$ is the matrix used to produce the decoupled key that carries Rotary Positional Embedding (RoPE) (Su et al., 2024); $\operatorname{RoPE}(\cdot)$ denotes the operation that applies RoPE matrices; and $[\cdot;\cdot]$ denotes concatenation. Note that for MLA, only the blue-boxed vectors (i.e., $\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{c}_{t}^{KV}$ and $\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{k}_{t}^{R}$) need to be cached during generation, which results in significantly reduced KV cache while maintaining performance comparable to standard Multi-Head Attention (MHA) (Vaswani et al., 2017).
$\mathbf{c}_{t}^{KV}\in\mathbb{R}^{d_{c}}$ 是键和值的压缩潜在向量； $d_{c}(\ll d_{h}n_{h})$ 表示 KV 压缩维度； $W^{DKV}\in\mathbb{R}^{d_{c}\times d}$ 表示下投影矩阵； $W^{UK},W^{UV}\in\mathbb{R}^{d_{h}n_{h}\times d_{c}}$ 分别是键和值的上投影矩阵； $W^{KR}\in\mathbb{R}^{d_{h}^{R}\times d}$ 是用于产生携带旋转位置嵌入（RoPE）的解耦键的矩阵（Su 等，2024）； $\operatorname{RoPE}(\cdot)$ 表示应用 RoPE 矩阵的操作； $[\cdot;\cdot]$ 表示拼接。请注意，对于 MLA，在生成过程中只需缓存蓝色框内的向量（即 $\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{c}_{t}^{KV}$ 和 $\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{k}_{t}^{R}$ ），这显著减少了 KV 缓存，同时保持了与标准多头注意力（MHA）（Vaswani 等，2017）相当的性能。

For the attention queries, we also perform a low-rank compression, which can reduce the activation memory during training:
对于关注查询，我们同样执行低秩压缩，这可以在训练期间减少激活内存：

where $\mathbf{c}_{t}^{Q}\in\mathbb{R}^{d_{c}^{\prime}}$ is the compressed latent vector for queries; $d_{c}^{\prime}(\ll d_{h}n_{h})$ denotes the query compression dimension; $W^{DQ}\in\mathbb{R}^{d_{c}^{\prime}\times d},W^{UQ}\in\mathbb{R}^{d_{h}n_{h}\times d_{c}^{\prime}}$ are the down-projection and up-projection matrices for queries, respectively; and $W^{QR}\in\mathbb{R}^{d_{h}^{R}n_{h}\times d_{c}^{\prime}}$ is the matrix to produce the decoupled queries that carry RoPE.

Ultimately, the attention queries ($\mathbf{q}_{t,i}$), keys ($\mathbf{k}_{j,i}$), and values ($\mathbf{v}_{j,i}^{C}$) are combined to yield the final attention output $\mathbf{u}_{t}$:

where $W^{O}\in\mathbb{R}^{d\times d_{h}n_{h}}$ denotes the output projection matrix.

For DeepSeek-V3, the communication overhead introduced by cross-node expert parallelism results in an inefficient computation-to-communication ratio of approximately 1:1. To tackle this challenge, we design an innovative pipeline parallelism algorithm called DualPipe, which not only accelerates model training by effectively overlapping forward and backward computation-communication phases, but also reduces the pipeline bubbles.
对于 DeepSeek-V3，跨节点专家并行引入的通信开销导致计算与通信比约为 1:1，效率低下。为了应对这一挑战，我们设计了一种创新的管道并行算法，称为 DualPipe，它不仅通过有效地重叠正向和反向计算-通信阶段来加速模型训练，还减少了管道气泡。

The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks. To be specific, we divide each chunk into four components: attention, all-to-all dispatch, MLP, and all-to-all combine. Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights, like in ZeroBubble (Qi et al., 2023b). In addition, we have a PP communication component. As illustrated in Figure 4, for a pair of forward and backward chunks, we rearrange these components and manually adjust the ratio of GPU SMs dedicated to communication versus computation. In this overlapping strategy, we can ensure that both all-to-all and PP communication can be fully hidden during execution. Given the efficient overlapping strategy, the full DualPipe scheduling is illustrated in Figure 5. It employs a bidirectional pipeline scheduling, which feeds micro-batches from both ends of the pipeline simultaneously and a significant portion of communications can be fully overlapped. This overlap also ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead.
双管的关键思想是在一对单独的前向和后向块中重叠计算和通信。具体来说，我们将每个块分为四个部分：注意力、全到全调度、MLP 和全到全组合。特别地，对于后向块，注意力和 MLP 进一步分为两部分，后向输入和后向权重，类似于 ZeroBubble（Qi 等人，2023b）。此外，我们还有一个 PP 通信组件。如图 4 所示，对于一对前向和后向块，我们重新排列这些组件，并手动调整用于通信和计算的 GPU SMs 的比率。在这种重叠策略中，我们可以确保全到全和 PP 通信在执行期间可以完全隐藏。鉴于高效的重叠策略，完整的 DualPipe 调度如图 5 所示。它采用双向管道调度，同时从管道两端提供微批次，并且大部分通信可以完全重叠。 这种重叠还确保了，随着模型进一步扩大，只要我们保持恒定的计算与通信比率，我们仍然可以在节点间使用细粒度专家，同时实现近乎零的全对全通信开销。

In addition, even in more general scenarios without a heavy communication burden, DualPipe still exhibits efficiency advantages. In Table 2, we summarize the pipeline bubbles and memory usage across different PP methods. As shown in the table, compared with ZB1P (Qi et al., 2023b) and 1F1B (Harlap et al., 2018), DualPipe significantly reduces the pipeline bubbles while only increasing the peak activation memory by $\frac{1}{PP}$ times. Although DualPipe requires keeping two copies of the model parameters, this does not significantly increase the memory consumption since we use a large EP size during training. Compared with Chimera (Li and Hoefler, 2021), DualPipe only requires that the pipeline stages and micro-batches be divisible by 2, without requiring micro-batches to be divisible by pipeline stages. In addition, for DualPipe, neither the bubbles nor activation memory will increase as the number of micro-batches grows.
此外，即使在更通用的场景中，没有沉重的通信负担，DualPipe 仍然显示出效率优势。在表 2 中，我们总结了不同 PP 方法中的管道气泡和内存使用情况。如表所示，与 ZB1P（Qi 等人，2023b）和 1F1B（Harlap 等人，2018）相比，DualPipe 显著减少了管道气泡，同时仅将峰值激活内存增加 $\frac{1}{PP}$ 倍。尽管 DualPipe 需要保留两个模型参数副本，但由于我们在训练期间使用了一个较大的 EP 大小，这并不会显著增加内存消耗。与 Chimera（Li 和 Hoefler，2021）相比，DualPipe 仅要求管道阶段和微批次可被 2 整除，而不要求微批次可被管道阶段整除。此外，对于 DualPipe，随着微批次数量的增加，气泡和激活内存都不会增加。

In order to ensure sufficient computational performance for DualPipe, we customize efficient cross-node all-to-all communication kernels (including dispatching and combining) to conserve the number of SMs dedicated to communication. The implementation of the kernels is co-designed with the MoE gating algorithm and the network topology of our cluster. To be specific, in our cluster, cross-node GPUs are fully interconnected with IB, and intra-node communications are handled via NVLink. NVLink offers a bandwidth of 160 GB/s, roughly 3.2 times that of IB (50 GB/s). To effectively leverage the different bandwidths of IB and NVLink, we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts, without being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink. This implies that, although DeepSeek-V3 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes $\times$ 3.2 experts/node) while preserving the same communication cost. Overall, under such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB and NVLink.
为确保 DualPipe 有足够的计算性能，我们定制了高效的跨节点全全连接通信内核（包括调度和组合），以节省用于通信的 SM 数量。内核的实现与 MoE 门控算法和我们的集群网络拓扑协同设计。具体来说，在我们的集群中，跨节点 GPU 通过 IB 完全互联，节点内通信通过 NVLink 处理。NVLink 提供 160 GB/s 的带宽，大约是 IB（50 GB/s）的 3.2 倍。为了有效地利用 IB 和 NVLink 的不同带宽，我们限制每个令牌最多分发到 4 个节点，从而减少 IB 流量。对于每个令牌，当其路由决策确定后，它将通过 IB 首先传输到目标节点上具有相同节点索引的 GPU。一旦到达目标节点，我们将努力确保它通过 NVLink 即时转发到托管其目标专家的特定 GPU，而不会被随后到达的令牌阻塞。 这种方式下，通过 IB 和 NVLink 的通信完全重叠，每个令牌可以高效地选择每个节点平均 3.2 位专家，而不会产生额外的 NVLink 开销。这意味着，尽管 DeepSeek-V3 实际上只选择了 8 位路由专家，但它可以将这个数字扩展到最多 13 位专家（4 节点 @ 0# 3.2 位专家/节点），同时保持相同的通信成本。总体而言，在这种通信策略下，只需要 20 个 SM 就可以充分利用 IB 和 NVLink 的带宽。

In detail, we employ the warp specialization technique (Bauer et al., 2014) and partition 20 SMs into 10 communication channels. During the dispatching process, (1) IB sending, (2) IB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps. The number of warps allocated to each communication task is dynamically adjusted according to the actual workload across all SMs. Similarly, during the combining process, (1) NVLink sending, (2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also handled by dynamically adjusted warps. In addition, both dispatching and combining kernels overlap with the computation stream, so we also consider their impact on other SM computation kernels. Specifically, we employ customized PTX (Parallel Thread Execution) instructions and auto-tune the communication chunk size, which significantly reduces the use of the L2 cache and the interference to other SMs.
详细来说，我们采用了变形专用技术（Bauer 等，2014）并将 20 个 SM 分为 10 个通信通道。在调度过程中，(1) IB 发送、(2) IB 到 NVLink 转发和(3) NVLink 接收由相应的变形处理。分配给每个通信任务变形的数量根据所有 SM 的实际工作负载动态调整。同样，在合并过程中，(1) NVLink 发送、(2) NVLink 到 IB 转发和累积以及(3) IB 接收和累积也由动态调整的变形处理。此外，调度和合并内核与计算流重叠，因此我们也考虑它们对其他 SM 计算内核的影响。具体来说，我们采用了定制的 PTX（并行线程执行）指令并自动调整通信块大小，这显著减少了 L2 缓存的占用和其他 SM 的干扰。

In order to reduce the memory footprint during training, we employ the following techniques.
为了在训练过程中减少内存占用，我们采用了以下技术。

Building upon widely adopted techniques in low-precision training (Kalamkar et al., 2019; Narang et al., 2017), we propose a mixed precision framework for FP8 training. In this framework, most compute-density operations are conducted in FP8, while a few key operations are strategically maintained in their original data formats to balance training efficiency and numerical stability. The overall framework is illustrated in Figure 6.

Firstly, in order to accelerate model training, the majority of core computation kernels, i.e., GEMM operations, are implemented in FP8 precision. These GEMM operations accept FP8 tensors as inputs and produce outputs in BF16 or FP32. As depicted in Figure 6, all three GEMMs associated with the Linear operator, namely Fprop (forward pass), Dgrad (activation backward pass), and Wgrad (weight backward pass), are executed in FP8. This design theoretically doubles the computational speed compared with the original BF16 method. Additionally, the FP8 Wgrad GEMM allows activations to be stored in FP8 for use in the backward pass. This significantly reduces memory consumption.

Despite the efficiency advantage of the FP8 format, certain operators still require a higher precision due to their sensitivity to low-precision computations. Besides, some low-cost operators can also utilize a higher precision with a negligible overhead to the overall training cost. For this reason, after careful investigations, we maintain the original precision (e.g., BF16 or FP32) for the following components: the embedding module, the output head, MoE gating modules, normalization operators, and attention operators. These targeted retentions of high precision ensure stable training dynamics for DeepSeek-V3. To further guarantee numerical stability, we store the master weights, weight gradients, and optimizer states in higher precision. While these high-precision components incur some memory overheads, their impact can be minimized through efficient sharding across multiple DP ranks in our distributed training system.

Based on our mixed precision FP8 framework, we introduce several strategies to enhance low-precision training accuracy, focusing on both the quantization method and the multiplication process.

In conjunction with our FP8 training framework, we further reduce the memory consumption and communication overhead by compressing cached activations and optimizer states into lower-precision formats.

The minimum deployment unit of the prefilling stage consists of 4 nodes with 32 GPUs. The attention part employs 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP), combined with 8-way Data Parallelism (DP8). Its small TP size of 4 limits the overhead of TP communication. For the MoE part, we use 32-way Expert Parallelism (EP32), which ensures that each expert processes a sufficiently large batch size, thereby enhancing computational efficiency. For the MoE all-to-all communication, we use the same method as in training: first transferring tokens across nodes via IB, and then forwarding among the intra-node GPUs via NVLink. In particular, we use 1-way Tensor Parallelism for the dense MLPs in shallow layers to save TP communication.

To achieve load balancing among different experts in the MoE part, we need to ensure that each GPU processes approximately the same number of tokens. To this end, we introduce a deployment strategy of redundant experts, which duplicates high-load experts and deploys them redundantly. The high-load experts are detected based on statistics collected during the online deployment and are adjusted periodically (e.g., every 10 minutes). After determining the set of redundant experts, we carefully rearrange experts among GPUs within a node based on the observed loads, striving to balance the load across GPUs as much as possible without increasing the cross-node all-to-all communication overhead. For the deployment of DeepSeek-V3, we set 32 redundant experts for the prefilling stage. For each GPU, besides the original 8 experts it hosts, it will also host one additional redundant expert. 

Furthermore, in the prefilling stage, to improve the throughput and hide the overhead of all-to-all and TP communication, we simultaneously process two micro-batches with similar computational workloads, overlapping the attention and MoE of one micro-batch with the dispatch and combine of another. 

Finally, we are exploring a dynamic redundancy strategy for experts, where each GPU hosts more experts (e.g., 16 experts), but only 9 will be activated during each inference step. Before the all-to-all operation at each layer begins, we compute the globally optimal routing scheme on the fly. Given the substantial computation involved in the prefilling stage, the overhead of computing this routing scheme is almost negligible. 

During decoding, we treat the shared expert as a routed one. From this perspective, each token will select 9 experts during routing, where the shared expert is regarded as a heavy-load one that will always be selected. The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. For the MoE part, each GPU hosts only one expert, and 64 GPUs are responsible for hosting redundant experts and shared experts. All-to-all communication of the dispatch and combine parts is performed via direct point-to-point transfers over IB to achieve low latency. Additionally, we leverage the IBGDA (NVIDIA, 2022) technology to further minimize latency and enhance communication efficiency. 

Similar to prefilling, we periodically determine the set of redundant experts in a certain interval, based on the statistical expert load from our online service. However, we do not need to rearrange experts since each GPU only hosts one expert. We are also exploring the dynamic redundancy strategy for decoding. However, this requires more careful optimization of the algorithm that computes the globally optimal routing scheme and the fusion with the dispatch kernel to reduce overhead. 

Additionally, to enhance throughput and hide the overhead of all-to-all communication, we are also exploring processing two micro-batches with similar computational workloads simultaneously in the decoding stage. Unlike prefilling, attention consumes a larger portion of time in the decoding stage. Therefore, we overlap the attention of one micro-batch with the dispatch+MoE+combine of another. In the decoding stage, the batch size per expert is relatively small (usually within 256 tokens), and the bottleneck is memory access rather than computation. Since the MoE part only needs to load the parameters of one expert, the memory access overhead is minimal, so using fewer SMs will not significantly affect the overall performance. Therefore, to avoid impacting the computation speed of the attention part, we can allocate only a small portion of SMs to dispatch+MoE+combine. 

In DeepSeek-V3, we implement the overlap between computation and communication to hide the communication latency during computation. This significantly reduces the dependency on communication bandwidth compared to serial computation and communication. However, the current communication implementation relies on expensive SMs (e.g., we allocate 20 out of the 132 SMs available in the H800 GPU for this purpose), which will limit the computational throughput. Moreover, using SMs for communication results in significant inefficiencies, as tensor cores remain entirely -utilized.

Currently, the SMs primarily perform the following tasks for all-to-all communication:

• •

  Forwarding data between the IB (InfiniBand) and NVLink domain while aggregating IB traffic destined for multiple GPUs within the same node from a single GPU.

  Report issue for preceding element
• •

  Transporting data between RDMA buffers (registered GPU memory regions) and input/output buffers.

  Report issue for preceding element
• •

  Executing reduce operations for all-to-all combine.

  Report issue for preceding element
• •

  Managing fine-grained memory layout during chunked data transferring to multiple experts across the IB and NVLink domain.

  Report issue for preceding element

We aspire to see future vendors developing hardware that offloads these communication tasks from the valuable computation unit SM, serving as a GPU co-processor or a network co-processor like NVIDIA SHARP Graham et al. (2016). Furthermore, to reduce application programming complexity, we aim for this hardware to unify the IB (scale-out) and NVLink (scale-up) networks from the perspective of the computation units. With this unified interface, computation units can easily accomplish operations such as read, write, multicast, and reduce across the entire IB-NVLink-unified domain via submitting communication requests based on simple primitives.

The base model of DeepSeek-V3 is pretrained on a multilingual corpus with English and Chinese constituting the majority, so we evaluate its performance on a series of benchmarks primarily in English and Chinese, as well as on a multilingual benchmark. Our evaluation is based on our internal evaluation framework integrated in our HAI-LLM framework. Considered benchmarks are categorized and listed as follows, where underlined benchmarks are in Chinese and double-underlined benchmarks are multilingual ones:

Multi-subject multiple-choice datasets include MMLU (Hendrycks et al., 2020), MMLU-Redux (Gema et al., 2024), MMLU-Pro (Wang et al., 2024b), MMMLU (OpenAI, 2024b), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023).

Language understanding and reasoning datasets include HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), and BigBench Hard (BBH) (Suzgun et al., 2022).

Closed-book question answering datasets include TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019).

Reading comprehension datasets include RACE Lai et al. (2017), DROP (Dua et al., 2019), C3 (Sun et al., 2019a), and CMRC (Cui et al., 2019).

Reference disambiguation datasets include CLUEWSC (Xu et al., 2020) and WinoGrande Sakaguchi et al. (2019).

Language modeling datasets include Pile (Gao et al., 2020).

Chinese understanding and culture datasets include CCPM (Li et al., 2021).

Math datasets include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MGSM (Shi et al., 2023), and CMath (Wei et al., 2023).

Code datasets include HumanEval (Chen et al., 2021), LiveCodeBench-Base (0801-1101) (Jain et al., 2024), MBPP (Austin et al., 2021), and CRUXEval (Gu et al., 2024).

Standardized exams include AGIEval (Zhong et al., 2023). Note that AGIEval includes both English and Chinese subsets.

Following our previous work (DeepSeek-AI, 2024b, c), we adopt perplexity-based evaluation for datasets including HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, MMLU-Redux, MMLU-Pro, MMMLU, ARC-Easy, ARC-Challenge, C-Eval, CMMLU, C3, and CCPM, and adopt generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, MGSM, HumanEval, MBPP, LiveCodeBench-Base, CRUXEval, BBH, AGIEval, CLUEWSC, CMRC, and CMath. In addition, we perform language-modeling-based evaluation for Pile-test and use Bits-Per-Byte (BPB) as the metric to guarantee fair comparison among models using different tokenizers.

In Table 3, we compare the base model of DeepSeek-V3 with the state-of-the-art open-source base models, including DeepSeek-V2-Base (DeepSeek-AI, 2024c) (our previous release), Qwen2.5 72B Base (Qwen, 2024b), and LLaMA-3.1 405B Base (AI@Meta, 2024b). We evaluate all these models with our internal evaluation framework, and ensure that they share the same evaluation setting. Note that due to the changes in our evaluation framework over the past months, the performance of DeepSeek-V2-Base exhibits a slight difference from our previously reported results. Overall, DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base, and surpasses LLaMA-3.1 405B Base in the majority of benchmarks, essentially becoming the strongest open-source model.

From a more detailed perspective, we compare DeepSeek-V3-Base with the other open-source base models individually. (1) Compared with DeepSeek-V2-Base, due to the improvements in our model architecture, the scale-up of the model size and training tokens, and the enhancement of data quality, DeepSeek-V3-Base achieves significantly better performance as expected. (2) Compared with Qwen2.5 72B Base, the state-of-the-art Chinese open-source model, with only half of the activated parameters, DeepSeek-V3-Base also demonstrates remarkable advantages, especially on English, multilingual, code, and math benchmarks. As for Chinese benchmarks, except for CMMLU, a Chinese multi-subject multiple-choice task, DeepSeek-V3-Base also shows better performance than Qwen2.5 72B. (3) Compared with LLaMA-3.1 405B Base, the largest open-source model with 11 times the activated parameters, DeepSeek-V3-Base also exhibits much better performance on multilingual, code, and math benchmarks. As for English and Chinese language benchmarks, DeepSeek-V3-Base shows competitive or better performance, and is especially good on BBH, MMLU-series, DROP, C-Eval, CMMLU, and CCPM.

Due to our efficient architectures and comprehensive engineering optimizations, DeepSeek-V3 achieves extremely high training efficiency. Under our training framework and infrastructures, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, which is much cheaper than training 72B or 405B dense models.

In Table 4, we show the ablation results for the MTP strategy. To be specific, we validate the MTP strategy on top of two baseline models across different scales. At the small scale, we train a baseline MoE model comprising 15.7B total parameters on 1.33T tokens. At the large scale, we train a baseline MoE model comprising 228.7B total parameters on 540B tokens. On top of them, keeping the training data and the other architectures the same, we append a 1-depth MTP module onto them and train two models with the MTP strategy for comparison. Note that during inference, we directly discard the MTP module, so the inference costs of the compared models are exactly the same. From the table, we can observe that the MTP strategy consistently enhances the model performance on most of the evaluation benchmarks.
在表 4 中，我们展示了 MTP 策略的消融结果。具体来说，我们在两个基线模型的基础上，针对不同规模进行了 MTP 策略的验证。在小规模上，我们在 1.33T 个标记上训练了一个包含 15.7B 个总参数的基线 MoE 模型。在大规模上，我们在 540B 个标记上训练了一个包含 228.7B 个总参数的基线 MoE 模型。在此基础上，保持训练数据和其它架构不变，我们在其上附加了一个 1 层深度的 MTP 模块，并使用 MTP 策略训练了两个模型以进行比较。注意，在推理过程中，我们直接丢弃了 MTP 模块，因此比较模型的推理成本完全相同。从表中可以看出，MTP 策略在大多数评估基准上持续提升了模型性能。

In Table 5, we show the ablation results for the auxiliary-loss-free balancing strategy. We validate this strategy on top of two baseline models across different scales. At the small scale, we train a baseline MoE model comprising 15.7B total parameters on 1.33T tokens. At the large scale, we train a baseline MoE model comprising 228.7B total parameters on 578B tokens. Both of the baseline models purely use auxiliary losses to encourage load balance, and use the sigmoid gating function with top-K affinity normalization. Their hyper-parameters to control the strength of auxiliary losses are the same as DeepSeek-V2-Lite and DeepSeek-V2, respectively. On top of these two baseline models, keeping the training data and the other architectures the same, we remove all auxiliary losses and introduce the auxiliary-loss-free balancing strategy for comparison. From the table, we can observe that the auxiliary-loss-free strategy consistently achieves better model performance on most of the evaluation benchmarks.

The key distinction between auxiliary-loss-free balancing and sequence-wise auxiliary loss lies in their balancing scope: batch-wise versus sequence-wise. Compared with the sequence-wise auxiliary loss, batch-wise balancing imposes a more flexible constraint, as it does not enforce in-domain balance on each sequence. This flexibility allows experts to better specialize in different domains. To validate this, we record and analyze the expert load of a 16B auxiliary-loss-based baseline and a 16B auxiliary-loss-free model on different domains in the Pile test set. As illustrated in Figure 9, we observe that the auxiliary-loss-free model demonstrates greater expert specialization patterns as expected.
辅助损失无平衡与序列辅助损失之间的关键区别在于它们的平衡范围：批量平衡与序列平衡。与序列辅助损失相比，批量平衡施加了更灵活的约束，因为它不对每个序列强制执行域内平衡。这种灵活性允许专家更好地专注于不同的领域。为了验证这一点，我们在 Pile 测试集的不同领域记录并分析了基于 16B 辅助损失的基线和 16B 辅助损失无模型的专业负载。如图 9 所示，我们观察到辅助损失无模型表现出预期的更大专家专业化模式。

To further investigate the correlation between this flexibility and the advantage in model performance, we additionally design and validate a batch-wise auxiliary loss that encourages load balance on each training batch instead of on each sequence. The experimental results show that, when achieving a similar level of batch-wise load balance, the batch-wise auxiliary loss can also achieve similar model performance to the auxiliary-loss-free method. To be specific, in our experiments with 1B MoE models, the validation losses are: 2.258 (using a sequence-wise auxiliary loss), 2.253 (using the auxiliary-loss-free method), and 2.253 (using a batch-wise auxiliary loss). We also observe similar results on 3B MoE models: the model using a sequence-wise auxiliary loss achieves a validation loss of 2.085, and the models using the auxiliary-loss-free method or a batch-wise auxiliary loss achieve the same validation loss of 2.080.
为了进一步研究这种灵活性与模型性能优势之间的关系，我们额外设计并验证了一种批量辅助损失，它鼓励在每个训练批次上实现负载均衡，而不是在每个序列上。实验结果表明，在实现类似的批量负载均衡水平时，批量辅助损失也能达到与无辅助损失方法相似的模型性能。具体来说，在我们的 1B MoE 模型实验中，验证损失为：2.258（使用序列辅助损失）、2.253（使用无辅助损失方法）和 2.253（使用批量辅助损失）。我们还在 3B MoE 模型上观察到类似的结果：使用序列辅助损失的模型达到的验证损失为 2.085，而使用无辅助损失方法或批量辅助损失的模型达到相同的验证损失 2.080。

In addition, although the batch-wise load balancing methods show consistent performance advantages, they also face two potential challenges in efficiency: (1) load imbalance within certain sequences or small batches, and (2) domain-shift-induced load imbalance during inference. The first challenge is naturally addressed by our training framework that uses large-scale expert parallelism and data parallelism, which guarantees a large size of each micro-batch. For the second challenge, we also design and implement an efficient inference framework with redundant expert deployment, as described in Section 3.4, to overcome it.
此外，尽管批量负载均衡方法表现出持续的性能优势，但它们也面临着两个潜在的效率挑战：（1）某些序列或小批次的负载不平衡，以及（2）推理过程中由领域迁移引起的负载不平衡。第一个挑战通过我们的训练框架自然解决，该框架使用大规模专家并行和数据并行，保证了每个微批次的较大规模。对于第二个挑战，我们还在第 3.4 节中设计并实现了一个高效的推理框架，具有冗余专家部署，以克服它。

We employ a rule-based Reward Model (RM) and a model-based RM in our RL process.
我们在我们强化学习过程中采用基于规则的奖励模型（RM）和基于模型的 RM。

Similar to DeepSeek-V2 (DeepSeek-AI, 2024c), we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which foregoes the critic model that is typically with the same size as the policy model, and estimates the baseline from group scores instead. Specifically, for each question $q$, GRPO samples a group of outputs $\{o_{1},o_{2},\cdots,o_{G}\}$ from the old policy model $\pi_{\theta_{old}}$ and then optimizes the policy model $\pi_{\theta}$ by maximizing the following objective:
类似于 DeepSeek-V2（DeepSeek-AI，2024c），我们采用组相对策略优化（GRPO）（Shao 等，2024），它放弃了与策略模型大小相同的评论家模型，并从组分数中估计基线。具体来说，对于每个问题 $q$ ，GRPO 从旧策略模型 $\pi_{\theta_{old}}$ 中采样一组输出 $\{o_{1},o_{2},\cdots,o_{G}\}$ ，然后通过最大化以下目标来优化策略模型 $\pi_{\theta}$ ：

where $\varepsilon$ and $\beta$ are hyper-parameters; $\pi_{ref}$ is the reference model; and $A_{i}$ is the advantage, derived from the rewards $\{r_{1},r_{2},\ldots,r_{G}\}$ corresponding to the outputs within each group:
$\varepsilon$ 和 $\beta$ 是超参数； $\pi_{ref}$ 是参考模型； $A_{i}$ 是从每个组内的输出对应的奖励 $\{r_{1},r_{2},\ldots,r_{G}\}$ 中导出的优势：

We incorporate prompts from diverse domains, such as coding, math, writing, role-playing, and question answering, during the RL process. This approach not only aligns the model more closely with human preferences but also enhances performance on benchmarks, especially in scenarios where available SFT data are limited.
我们将在 RL 过程中整合来自不同领域的提示，例如编码、数学、写作、角色扮演和问答。这种方法不仅使模型更接近人类偏好，而且提高了在基准测试中的性能，尤其是在可用 SFT 数据有限的情况下。

Table 6 presents the evaluation results, showcasing that DeepSeek-V3 stands as the best-performing open-source model. Additionally, it is competitive against frontier closed-source models like GPT-4o and Claude-3.5-Sonnet.
表 6 展示了评估结果，显示 DeepSeek-V3 是表现最佳的开放源代码模型。此外，它在与 GPT-4o 和 Claude-3.5-Sonnet 等前沿闭源模型竞争中表现良好。

In addition to standard benchmarks, we also evaluate our models on open-ended generation tasks using LLMs as judges, with the results shown in Table 7. Specifically, we adhere to the original configurations of AlpacaEval 2.0 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024a), which leverage GPT-4-Turbo-1106 as judges for pairwise comparisons. On Arena-Hard, DeepSeek-V3 achieves an impressive win rate of over 86% against the baseline GPT-4-0314, performing on par with top-tier models like Claude-Sonnet-3.5-1022. This underscores the robust capabilities of DeepSeek-V3, especially in dealing with complex prompts, including coding and debugging tasks. Furthermore, DeepSeek-V3 achieves a groundbreaking milestone as the first open-source model to surpass 85% on the Arena-Hard benchmark. This achievement significantly bridges the performance gap between open-source and closed-source models, setting a new standard for what open-source models can accomplish in challenging domains.
除了标准基准测试外，我们还使用LLMs作为评委，评估我们的模型在开放式生成任务上的表现，结果如表 7 所示。具体来说，我们遵循 AlpacaEval 2.0（Dubois 等人，2024 年）和 Arena-Hard（Li 等人，2024a）的原始配置，这些配置利用 GPT-4-Turbo-1106 作为评委进行成对比较。在 Arena-Hard 上，DeepSeek-V3 实现了超过 86%的惊人胜率，与 Claude-Sonnet-3.5-1022 等顶级模型表现相当。这突显了 DeepSeek-V3 的强大能力，尤其是在处理复杂提示，包括编码和调试任务方面。此外，DeepSeek-V3 作为第一个在 Arena-Hard 基准测试中超过 85%的开源模型，实现了突破性的里程碑。这一成就显著缩小了开源模型和闭源模型之间的性能差距，为开源模型在具有挑战性的领域所能取得的成就设定了新的标准。

Similarly, DeepSeek-V3 showcases exceptional performance on AlpacaEval 2.0, outperforming both closed-source and open-source models. This demonstrates its outstanding proficiency in writing tasks and handling straightforward question-answering scenarios. Notably, it surpasses DeepSeek-V2.5-0905 by a significant margin of 20%, highlighting substantial improvements in tackling simple tasks and showcasing the effectiveness of its advancements.
同样，DeepSeek-V3 在 AlpacaEval 2.0 上展现出卓越的性能，优于闭源和开源模型。这证明了它在写作任务和简单问答场景处理方面的出色能力。值得注意的是，它比 DeepSeek-V2.5-0905 高出 20%，显著提升了处理简单任务的能力，并展示了其进步的有效性。

We compare the judgment ability of DeepSeek-V3 with state-of-the-art models, namely GPT-4o and Claude-3.5. Table 8 presents the performance of these models in RewardBench (Lambert et al., 2024). DeepSeek-V3 achieves performance on par with the best versions of GPT-4o-0806 and Claude-3.5-Sonnet-1022, while surpassing other versions. Additionally, the judgment ability of DeepSeek-V3 can also be enhanced by the voting technique. Therefore, we employ DeepSeek-V3 along with voting to offer self-feedback on open-ended questions, thereby improving the effectiveness and robustness of the alignment process.
我们比较了 DeepSeek-V3 与最先进的模型 GPT-4o 和 Claude-3.5 的判断能力。表 8 展示了这些模型在 RewardBench（Lambert 等人，2024）上的性能。DeepSeek-V3 的性能与 GPT-4o-0806 和 Claude-3.5-Sonnet-1022 的最佳版本相当，并超越了其他版本。此外，DeepSeek-V3 的判断能力还可以通过投票技术得到增强。因此，我们采用 DeepSeek-V3 和投票技术来对开放式问题提供自我反馈，从而提高对齐过程的有效性和鲁棒性。

We ablate the contribution of distillation from DeepSeek-R1 based on DeepSeek-V2.5. The baseline is trained on short CoT data, whereas its competitor uses data generated by the expert checkpoints described above.
我们基于 DeepSeek-V2.5 消除了 DeepSeek-R1 中蒸馏的贡献。基线是在短 CoT 数据上训练的，而其竞争对手使用上述专家检查点生成数据。

Table 9 demonstrates the effectiveness of the distillation data, showing significant improvements in both LiveCodeBench and MATH-500 benchmarks. Our experiments reveal an interesting trade-off: the distillation leads to better performance but also substantially increases the average response length. To maintain a balance between model accuracy and computational efficiency, we carefully selected optimal settings for DeepSeek-V3 in distillation.
表 9 展示了蒸馏数据的有效性，显示了 LiveCodeBench 和 MATH-500 基准测试中的显著改进。我们的实验揭示了一个有趣的权衡：蒸馏带来了更好的性能，但也显著增加了平均响应长度。为了在模型准确性和计算效率之间保持平衡，我们在蒸馏中仔细选择了 DeepSeek-V3 的最佳设置。

Our research suggests that knowledge distillation from reasoning models presents a promising direction for post-training optimization. While our current work focuses on distilling data from mathematics and coding domains, this approach shows potential for broader applications across various task domains. The effectiveness demonstrated in these specific areas indicates that long-CoT distillation could be valuable for enhancing model performance in other cognitive tasks requiring complex reasoning. Further exploration of this approach across different domains remains an important direction for future research.
我们的研究表明，从推理模型中进行知识蒸馏是后训练优化的一个有希望的方向。虽然我们目前的工作集中在从数学和编码领域提取数据，但这种方法在各个任务领域具有广泛的应用潜力。在这些特定领域所展示的有效性表明，长 CoT 蒸馏对于增强其他需要复杂推理的认知任务中的模型性能可能是有价值的。在未来研究中，跨不同领域进一步探索这种方法仍然是一个重要的方向。

Rewards play a pivotal role in RL, steering the optimization process. In domains where verification through external tools is straightforward, such as some coding or mathematics scenarios, RL demonstrates exceptional efficacy. However, in more general scenarios, constructing a feedback mechanism through hard coding is impractical. During the development of DeepSeek-V3, for these broader contexts, we employ the constitutional AI approach (Bai et al., 2022), leveraging the voting evaluation results of DeepSeek-V3 itself as a feedback source. This method has produced notable alignment effects, significantly enhancing the performance of DeepSeek-V3 in subjective evaluations. By integrating additional constitutional inputs, DeepSeek-V3 can optimize towards the constitutional direction. We believe that this paradigm, which combines supplementary information with LLMs as a feedback source, is of paramount importance. The LLM serves as a versatile processor capable of transforming unstructured information from diverse scenarios into rewards, ultimately facilitating the self-improvement of LLMs. Beyond self-rewarding, we are also dedicated to uncovering other general and scalable rewarding methods to consistently advance the model capabilities in general scenarios.
奖励在强化学习中扮演关键角色，引导优化过程。在验证通过外部工具简单易行的领域，例如某些编码或数学场景，强化学习表现出卓越的效率。然而，在更普遍的场景中，通过硬编码构建反馈机制是不切实际的。在开发 DeepSeek-V3 期间，对于这些更广泛的背景，我们采用宪法 AI 方法（Bai 等，2022），利用 DeepSeek-V3 自身的投票评估结果作为反馈来源。这种方法产生了显著的协调效应，显著提高了 DeepSeek-V3 在主观评估中的性能。通过整合额外的宪法输入，DeepSeek-V3 可以优化向宪法方向。我们相信，这种结合补充信息与LLMs作为反馈来源的范式至关重要。LLM作为一个多功能的处理器，能够将来自不同场景的非结构化信息转化为奖励，最终促进LLMs的自我提升。 除了自我奖励外，我们还致力于发现其他通用且可扩展的奖励方法，以持续提升模型在一般场景下的能力。

Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique. Combined with the framework of speculative decoding (Leviathan et al., 2023; Xia et al., 2023), it can significantly accelerate the decoding speed of the model. A natural question arises concerning the acceptance rate of the additionally predicted token. Based on our evaluation, the acceptance rate of the second token prediction ranges between 85% and 90% across various generation topics, demonstrating consistent reliability. This high acceptance rate enables DeepSeek-V3 to achieve a significantly improved decoding speed, delivering 1.8 times TPS (Tokens Per Second).
DeepSeek-V3 通过 MTP 技术预测下一个 2 个 token，而不是仅仅预测下一个单个 token。结合推测解码框架（Leviathan 等人，2023；Xia 等人，2023），它可以显著提高模型的解码速度。关于额外预测 token 的接受率，自然会提出一个自然的问题。根据我们的评估，第二个 token 预测的接受率在各个生成主题中介于 85%到 90%之间，显示出一致的可靠性。这种高接受率使 DeepSeek-V3 能够实现解码速度的显著提高，达到每秒 1.8 倍的 TPS（token 每秒）。