Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for
Large Language Model Training
Seq1F1B：高效的序列级流水线并行 大型语言模型训练

As shown in Figure 1, 1F1B includes three phases to train a batch of sequences: warm-up phase, steady phase, and cooling-down phase. Given $P$ devices (e.g., GPUs) to perform a 1F1B schedule to train $M$ micro-batches, with each device responsible for one pipeline stage, the size of PP is $P$. After splitting the batch into $M$ micro-batches, during the warm-up phase, each device executes the forward passes of the first few micro-batches, and the number of forward passes $w_{i}$ executed by the $i$-th device is
如图 1所示，1F1B包括三个阶段来训练一批序列：热身阶段、稳定阶段和冷却阶段。 给定 $P$ 个设备（例如，GPU）执行 1F1B 调度来训练 $M$ 个微批次，每个设备负责一个流水线阶段，PP 的大小为 $P$ 。 在将批次分成 $M$ 个微批次后，在热身阶段，每个设备执行前几个微批次的前向传递，第 $i$ 个设备执行的前向传递数量为 $w_{i}$

When $M\leq P$, 1F1B degrades to the behavior of GPipe and does not process the steady phase. Otherwise, during the warm-up phase, a device responsible for an earlier stage performs one more forward pass than the device responsible for its subsequent stage. Each forward pass results in intermediate states enqueued in a FIFO queue $Q$ to be used later for the gradient computation of backward passes.
当 $M\leq P$ 时，1F1B 退化为 GPipe 的行为，并且不处理稳定阶段。否则，在热身阶段，负责较早阶段的设备比负责其后续阶段的设备多执行一次前向传递。每次前向传递都会在 FIFO 队列 $Q$ 中排入中间状态，以便稍后用于后向传递的梯度计算。

During the steady phase, each device performs one forward pass and enqueues the resulting intermediate states into $Q$. After a device executes a forward pass, the device dequeues specific intermediate states from $Q$ and immediately executes a backward pass for gradient computation, where the “1F1B” name comes from. Note that, the bubble ratio is minimal during the steady phase, and the number of 1F1B passes in the steady phase is given by $M-\text{w}_{i}$. As $M$ increases, the proportion of the steady phase in the entire pipeline increases, which reduces the bubble ratio. After the steady phase, the 1F1B schedule enters the cooling-down phase, which is symmetric to the warm-up phase and involves executing the same number of backward passes as in the warm-up phase.
在稳定阶段，每个设备执行一次前向传递，并将生成的中间状态排入 $Q$ 。在设备执行前向传递后，该设备从 $Q$ 中出队特定的中间状态，并立即执行后向传递以进行梯度计算，这就是“1F1B”名称的由来。 请注意，在稳定阶段，气泡比率最小，稳定阶段中的 1F1B 传递次数由 $M-\text{w}_{i}$ 给出。 随着 $M$ 的增加，整个流水线中稳定阶段的比例增加，从而降低了气泡比率。 在稳定阶段之后，1F1B 调度进入冷却阶段，该阶段与热身阶段对称，并涉及执行与热身阶段相同数量的反向传递。

The primary optimization of 1F1B is to ensure that the memory consumption of intermediate states is independent of $M$. The peak memory consumption for intermediate states is determined by the number of items in the queue $Q$ at the end of the warm-up phase, where each device holds $w_{i}$ intermediate states. Assuming the total memory consumption of intermediate states is $A$, the peak memory consumption of the $i$-th device is $\frac{A\times w_{i}}{\sum_{j=1}^{P}w_{j}}$. During the steady and cooling-down phases, this consumption does not increase since each backward pass frees the storage space for its associated intermediate states
1F1B 的主要优化是确保中间状态的内存消耗与 $M$ 无关。中间状态的峰值内存消耗由队列 $Q$ 末尾的项目数量决定 在热身阶段，每个设备持有 $w_{i}$ 个中间状态。假设总的 中间状态的内存消耗为 $A$ ，第 $i$ 个设备的峰值内存消耗为 $\frac{A\times w_{i}}{\sum_{j=1}^{P}w_{j}}$ 。在稳定和冷却阶段， 这种消耗不会增加，因为每次反向传递都会释放其相关中间状态的存储空间。

Language modeling is the most common unsupervised objective in training LLMs. In language modeling, each token is predicted sequentially while conditioned on the preceding tokens, embodying the principles of sequential generation, as formulated as
语言建模是训练 LLM 中最常见的无监督目标。在语言建模中，每个标记在条件于前面的标记的情况下按顺序预测，体现了顺序生成的原则，具体表述为

where $T$ is the sequence length. In the context of language modeling using Transformers, the causal attention mechanism ensures that each token in a sequence can only see its predecessors to process hidden states, including itself. Given a sequence of input token states $\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}\}$, the output of the attention mechanism for each token can be computed as follows. Each token states $\mathbf{x}_{i}$ is mapped into a query vector $\mathbf{q}_{i}$, a key vector $\mathbf{k}_{i}$, and a value vector $\mathbf{v}_{i}$, and output for each token $\mathbf{o}_{i}$ is computed by attending over all previous tokens as follows,
其中 $T$ 是序列长度。在使用 Transformers 进行语言建模的背景下，因果注意机制确保序列中的每个标记只能看到其前面的标记以处理隐藏状态，包括它自己。 给定一系列输入标记状态 $\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}\}$ ，每个标记的注意机制输出可以如下计算。每个标记状态 $\mathbf{x}_{i}$ 被映射为查询向量 $\mathbf{q}_{i}$ 、键向量 $\mathbf{k}_{i}$ 和值向量 $\mathbf{v}_{i}$ ，每个标记 $\mathbf{o}_{i}$ 的输出通过关注所有先前的标记来计算，如下所示，

where $d_{k}$ is the vector dimension. Based on these characteristics, it is clear that to partition the Transformer computation across the sequence dimension must retain the key and value vectors of all preceding tokens. The forward and backward passes also need to maintain a specific order. The forward pass of each token must follow the completion of its predecessor’s computation, while the backward pass requires the subsequent token’s gradients to complete its computation. This computational dependency needs to be fully considered in sequence-level pipeline schedule.
其中 $d_{k}$ 是向量维度。 基于这些特性，很明显要在序列维度上划分 Transformer 计算必须保留所有前面标记的键和值向量。 前向和反向传递也需要保持特定顺序。每个标记的前向传递必须在其前一个标记的计算完成后进行，而反向传递需要后续标记的梯度来完成其计算。 这种计算依赖性需要在序列级流水线调度中充分考虑。

From Figure 1, we observe that the original 1F1B schedule cannot accommodate the splitting of micro-batches along the sequence dimension because the last stage needs to immediately execute a backward pass after forwarding a micro-batch. A straightforward adaptation method is to divide each original 1F1B micro-batch into $k$ segments and then execute a $k$F$b$B pipeline  (Li et al., 2021). Although this schedule can reduce some bubbles in 1F1B, it does not save memory usage.
从图 1可以看出，原始的 1F1B 调度无法适应沿序列维度拆分微批次，因为最后一个阶段需要在转发微批次后立即执行反向传递。一种简单的适应方法是将每个原始 1F1B 微批次分为 $k$ 个部分，然后执行 $k$ F $b$ B 流水线 (Li et al., 2021)。虽然这种调度可以减少 1F1B 中的一些气泡，但不能节省内存使用。

To achieve a more efficient sequence-level 1F1B pipeline schedule, we propose Seq1F1B. Similar to 1F1B, the schedule of Seq1F1B is also divided into three phases: warm-up phase, steady phase, and cooling-down phase. During the warm-up phase, the number of sub-sequences of the $i$-th device is computed according to
为了实现更高效的序列级 1F1B 流水线调度，我们提出了 Seq1F1B。 类似于 1F1B，Seq1F1B 的调度也分为三个阶段：热身阶段、稳定阶段和冷却阶段。 在热身阶段， $i$ 设备的子序列数量根据以下公式计算

where $P$ is the size of the PP and $k$ indicates the number of divisions of the sequence. This equation ensures that the last stage can perform a backward pass on the last sub-sequence of the first micro-batch when entering the steady phase, and the device responsible for each stage performs one more forward pass than the device responsible for the subsequent stage. Here, we construct a partially ordered queue $Q_{s}$, where each pop returns the tail sequence from the earliest enqueued intermediate states. This satisfies the FIFO principle in the batch dimension and the first-in-last-out (FILO) principle in the sequence dimension. In each step of the warm-up phase, devices execute one forward pass and enqueue the corresponding intermediate states of sub-sequences into $Q_{s}$. During the steady phase, after each device completes a forward pass, it dequeues intermediate states from $Q_{s}$ and performs a backward pass, following the standard 1F1B process, except that the units for forward and backward passes become a sub-sequence. During the cooling-down phase, devices dequeue the remaining intermediate states from $Q_{s}$, perform backward passes for remaining subsequences and accumulate the gradient to ensure mathematical equivalent with other synchronous schedules.
其中 $P$ 是 PP 的大小， $k$ 表示 序列的划分数量。 这个方程确保在进入稳定阶段时，最后一个阶段可以对第一个微批次的最后一个子序列执行反向传播，并且负责每个阶段的设备比负责后续阶段的设备多执行一次前向传播。 在这里，我们构建了一个部分有序队列 $Q_{s}$ ，每次弹出操作返回最早入队的中间状态的尾部序列。这在批次维度上满足 FIFO 原则，在序列维度上满足先进后出（FILO）原则。在预热阶段的每一步中，设备执行一次前向传播，并将子序列的相应中间状态入队到 $Q_{s}$ 。 在稳定阶段，每个设备完成一次前向传播后，从 $Q_{s}$ 中出队中间状态并执行反向传播，遵循标准的 1F1B 过程，只是前向和反向传播的单位变成了一个子序列。 在冷却阶段，设备从 $Q_{s}$ 中出队剩余的中间状态，对剩余的子序列执行反向传播并累积梯度，以确保与其他同步调度在数学上等效。

From the timeline shown in Figure 1, it is evident that the Seq1F1B schedule offers a shorter execution time and significantly fewer bubbles, compared to the original 1F1B schedule. Meanwhile, it can be seen that each device now has less memory consumption since the sub-sequence is smaller than the micro-batch. Another observation is that optimizations similar to ZB1P can also be applied to Seq1F1B by delaying the gradient computation associated with weights in the backward pass. For more details, we refer to our appendix.
从图1所示的时间线上可以看出，Seq1F1B 调度相比原始的 1F1B 调度提供了更短的执行时间和显著更少的气泡。同时可以看到，由于子序列比微批次小，每个设备现在的内存消耗更少。另一个观察是，类似于 ZB1P 的优化也可以通过延迟与反向传播中权重相关的梯度计算应用于 Seq1F1B。更多细节请参阅我们的附录。

As shown in Figure 2, 1F1B-I (Narayanan et al., 2021b) achieves better efficiency by modifying the 1F1B schedule to support interleaved stages among devices. In 1F1B-I, each device is assigned multiple stages. Suppose we have $P$ devices and $V$ stages $\{s_{1},s_{2},\ldots,s_{V}\}$ in our pipeline, where $V$ is a multiple of $P$. The $i$-th device will handle $n$ stages $\{s_{i},s_{i+P},s_{i+2P},\ldots,s_{i+(n-1)P}\}$, where $n=\frac{V}{P}$. The number of warm-up micro-batches of each device $i$ in 1F1B-I is as follows,
如图 2所示，1F1B-I (Narayanan et al., 2021b)通过修改 1F1B 调度以支持设备间的交错阶段来实现更高的效率。在 1F1B-I 中，每个设备被分配多个阶段。假设我们的流水线中有 $P$ 个设备和 $V$ 个阶段 $\{s_{1},s_{2},\ldots,s_{V}\}$ ，其中 $V$ 是 $P$ 的倍数。第 $i$ 个设备将处理 $n$ 个阶段 $\{s_{i},s_{i+P},s_{i+2P},\ldots,s_{i+(n-1)P}\}$ ，其中 $n=\frac{V}{P}$ 。1F1B-I 中每个设备的预热微批次数量 $i$ 如下，

After completing $P$ iterations of forward and backward passes, each device switches its context to the next stage that the device is responsible for. From Figure 2, the above part shows a 1F1B-I pipeline with $P=4$ and $V=8$, in which each device handles 2 stages. The 1F1B-I schedule reduces the bubble ratio by interleaving stages among devices. However, this interleaving slightly increases memory consumption, as the number of warm-up micro-batches $w_{i}$ is greater than that of 1F1B.
完成 $P$ 次前向和反向传播迭代后，每个设备切换其上下文到设备负责的下一个阶段。从图 2可以看出，上述部分展示了一个具有 $P=4$ 和 $V=8$ 的 1F1B-I 流水线，其中每个设备处理 2 个阶段。1F1B-I 调度通过在设备间交错阶段来减少气泡比率。然而，这种交错略微增加了内存消耗，因为预热微批次数量 $w_{i}$ 大于 1F1B。

Similar to 1F1B-I, Seq1F1B-I further modifies 1F1B-I to achieve a sequence-level schedule. From Figure 2, Seq1F1B-I effectively reduces pipeline bubbles and the memory footprint of intermediate states compared to 1F1B-I. Seq1F1B-I defines the number of warm-up sub-sequences as
类似于 1F1B-I，Seq1F1B-I 进一步修改 1F1B-I 以实现序列级调度。从图2可以看出，Seq1F1B-I 有效减少了流水线气泡和中间状态的内存占用，相比于 1F1B-I。Seq1F1B-I 定义了预热子序列的数量为

where $P$ is the size of the PP and $k$ indicates the number of divisions of the sequence. Using the partially ordered queue, Seq1F1B-I maintains a strict order of forward and backward passes as well as ensures the consistent semantics of gradient updates. From the perspective of pipeline bubbles, Seq1F1B-I outperforms both Seq1F1B and 1F1B-I. Besides, Seq1F1B-I requires slightly more memory than Seq1F1B but significantly less than 1F1B-I.
其中 $P$ 是 PP 的大小， $k$ 表示 序列的划分数量。 使用部分有序队列，Seq1F1B-I 保持了前向和反向传播的严格顺序，并确保梯度更新的一致语义。 从流水线气泡的角度来看，Seq1F1B-I 优于 Seq1F1B 和 1F1B-I。此外，Seq1F1B-I 需要的内存略多于 Seq1F1B，但显著少于 1F1B-I。

In this section, we detail the strategy of sequence partition and workload balance consideration. Previous works, such as  (Li et al., 2021), have discussed strategies for sequence partitioning. To achieve an efficient pipeline schedule, the processing cost for each sub-sequence must be approximately equal to avoid pipeline bubbles. To this end, we design a computation-wise partition strategy by estimating the FLOPs of sequences and constructing a theoretical solution aiming to make the FLOPs of all sub-sequences as closely as possible. For a input sequence $S=\{x_{1},x_{2},\cdots,x_{n}\}$, we devide it into $k$ segments $S=\{S_{1},S_{2},\cdots,S_{k}\}$. Each segment has a length of $n_{i}$, where $\sum_{i=1}^{k}n_{i}=n$. We expect the computational amount of each segment to be roughly the same, that is
在本节中，我们详细介绍了序列划分策略和工作负载平衡的考虑。先前的工作，如 (Li et al., 2021)，已经讨论了序列划分的策略。为了实现高效的流水线调度，每个子序列的处理成本必须大致相等，以避免流水线气泡。为此，我们通过估算序列的 FLOPs，设计了一种计算量划分策略，并构建了一个理论解决方案，旨在使所有子序列的 FLOPs 尽可能接近。 对于输入序列 $S=\{x_{1},x_{2},\cdots,x_{n}\}$ ，我们将其划分为 $k$ 段 $S=\{S_{1},S_{2},\cdots,S_{k}\}$ 。 每段的长度为 $n_{i}$ ，其中 $\sum_{i=1}^{k}n_{i}=n$ 。 我们期望每段的计算量大致相同，即

Specifically, we use the method proposed in (Hoffmann et al., 2022) to estimate the FLOPs for each subsequence, formulated as
具体来说，我们使用(Hoffmann et al., 2022)中提出的方法来估算每个子序列的 FLOPs，公式为

in which, $L$ is the number of layers, $d$ is the dimension of the model, and $P$ is the total number of parameters in the model. We have $k$ variables in Eq. (8) and $k$ equations in Eq. (7), and thus we can set up the equation to get the optimal segmentation.
其中， $L$ 是层数， $d$ 是模型的维度， $P$ 是模型中的参数总数。 我们在方程 (8)中有 $k$ 个变量，在方程 (7)中有 $k$ 个方程，因此我们可以建立方程以获得最佳分段。

In experiments, we measure Seq1F1B, Seq1F1B-I, 1F1B, and 1F1B-I under variable sequence lengths, different numbers of micro-batches, different numbers of GPUs, and different PP and TP sizes. Compared methods are as follows:
在实验中，我们测量了在不同序列长度、不同微批次数量、不同 GPU 数量以及不同 PP 和 TP 大小下的 Seq1F1B、Seq1F1B-I、1F1B 和 1F1B-I。比较的方法如下：

(1) Seq1F1B: Seq1F1B with computation-wise sequence partition strategy.
(1) Seq1F1B：采用计算序列分区策略的 Seq1F1B。

(2) Seq1F1B-I: Seq1F1B with interleaved stages and computation-wise sequence partition strategy.
(2) Seq1F1B-I：采用交错阶段和计算序列分区策略的 Seq1F1B。

(3) 1F1B/1F1B-I: 1F1B and 1F1B with interleaved stages in Megatron implementation.
(3) 1F1B/1F1B-I：Megatron 实现中的 1F1B 和采用交错阶段的 1F1B。

(4) Seq1F1B w/o cwp: Seq1F1B without computation-wise sequence partition strategy.
(4) Seq1F1B 无 cwp：不采用计算序列分区策略的 Seq1F1B。

(5) Seq1F1B-I w/o cwp: Seq1F1B-I without computation-wise sequence partition strategy.
(5) Seq1F1B-I 无 cwp：不采用计算序列分区策略的 Seq1F1B-I。

All assessments are based on the GPT model and model configurations are listed in Table 1. All experiments focus on long-sequence training since a lot of work has mentioned its importance. For hyperparameter configurations, we set the number of sequence splits to four and each device manages two stages in interleaved settings. Our implementation is based on the open-source Megatron-LM project (Narayanan et al., 2021b) and ensures reproducibility. We adopt Megatron-V3 (Korthikanti et al., 2023)’s tensor parallelism in all experiments since it is necessary for long sequence training.
所有评估均基于 GPT 模型，模型配置列在表格1中。所有实验都集中在长序列训练上，因为许多工作已提到其重要性。对于超参数配置，我们将序列分割数设置为四，每个设备在交错设置中管理两个阶段。我们的实现基于开源的 Megatron-LM 项目(Narayanan et al., 2021b)，确保可重复性。我们在所有实验中采用 Megatron-V3(Korthikanti et al., 2023)的张量并行性，因为它对于长序列训练是必要的。

Our experiments include three cluster settings: 1) 1 node with 8 NVIDIA A100 SXM 80G GPUs interconnected by NvLink. 2) 4 nodes interconnected by a RoCE RDMA network and each node has 8 NVIDIA A100 SXM 80G GPUs interconnected by NvLink. 3) 8 nodes interconnected by a RoCE RDMA network and each node has 8 NVIDIA A100 SXM 80G GPUs interconnected by NvLink. Each measurement in the experiment is repeated 100 times, and the standard deviation is recorded.
我们的实验包括三种集群设置：1）1 个节点，配备 8 个通过 NvLink 互连的 NVIDIA A100 SXM 80G GPU。2）4 个节点通过 RoCE RDMA 网络互连，每个节点有 8 个通过 NvLink 互连的 NVIDIA A100 SXM 80G GPU。3）8 个节点通过 RoCE RDMA 网络互连，每个节点有 8 个通过 NvLink 互连的 NVIDIA A100 SXM 80G GPU。实验中的每次测量重复 100 次，并记录标准差。

In Figure 3, we compared the memory consumption of our method with that of 1F1B and 1F1B-I. As can be seen, our method consistently requires less memory across all settings. Notably, it can support training a 30B model on a 64$\times$ A100 cluster, which is impossible for the traditional combination of PP and TP. Additionally, we recorded TFLOPS (teraFLOPS) per GPU in our experiments to measure the hardware utilization of different methods. From Table 2, 3, 4 and 5, our method Seq1F1B outperforms 1F1B and 1F1B-I under almost all settings in both training throughput and teraFLOPS.
在图 3中，我们将我们的方法与 1F1B 和 1F1B-I 的内存消耗进行了比较。可以看出，我们的方法在所有设置中始终需要更少的内存。值得注意的是，它可以支持在 64 $\times$ A100 集群上训练 30B 模型，而传统的 PP 和 TP 组合无法实现。此外，我们在实验中记录了每个 GPU 的 TFLOPS（teraFLOPS），以衡量不同方法的硬件利用率。从表 2、3、4 和 5中可以看出，我们的方法 Seq1F1B 在几乎所有设置下的训练吞吐量和 teraFLOPS 上都优于 1F1B 和 1F1B-I。

However, as observed in Table 3, 4, and 5, the Seq1F1B-I may have a performance degradation under multi-node settings. This could be due to the overly fine-grained interleaving of stage partitioning and input sequence partitioning, which also implies that more communication calls in TP (although the total communication volume remains unchanged), potentially leads to a decrease in performance. Another observation is that the efficiency of Seq1F1B becomes more pronounced as the sequence length increases. This is because the computation time for each micro-sequence extends with longer sequences, thereby enhancing the benefits derived from sequence partitioning.
然而，如表 3、4 和 5 所示，Seq1F1B-I 在多节点设置下可能会出现性能下降。这可能是由于阶段分区和输入序列分区的过于细粒度交错，这也意味着 TP 中更多的通信调用（尽管总通信量保持不变），可能导致性能下降。另一个观察是，随着序列长度的增加，Seq1F1B 的效率变得更加明显。这是因为较长序列的每个微序列的计算时间延长，从而增强了从序列分区中获得的好处。

We also conducted all experiments using Seq1F1B without computation-wise partitioning (Seq1F1B w/o cwp) and Seq1F1B-I without computation-wise partitioning (Seq1F1B-I w/o cwp) to evaluate the effectiveness of our computation-wise partition strategy. Under identical settings, employing the computation-wise partition strategy leads to performance enhancements ranging from approximately 10-30% for Seq1F1B compared to simply splitting the sequence.
我们还进行了所有实验，使用没有计算分区的 Seq1F1B（Seq1F1B w/o cwp）和没有计算分区的 Seq1F1B-I（Seq1F1B-I w/o cwp），以评估我们的计算分区策略的有效性。在相同设置下，采用计算分区策略相比于简单的序列分割，Seq1F1B 的性能提升约为 10-30%。

Across all experimental scales, Seq1F1B consistently surpassed Seq1F1B w/o cwp in performance. Table 6 highlights the ablation performance for a 2.7B model with a sequence length of 32k, demonstrating a performance boost of approximately 28% due to the computation-wise partitioning.
在所有实验规模中，Seq1F1B 的性能始终优于 Seq1F1B w/o cwp。表 6 突出了序列分区策略的消融性能，针对序列长度为 32k 的 2.7B 模型，计算分区带来了约 28% 的性能提升。