Understanding LLMs: A Comprehensive Overview from Training to Inference
理解LLMs：从训练到推理的全面概述

Self-Attention Structure [6]: Essentially, the attention mechanism aims at selecting a small amount of important information from a large amount of data and focusing on these important pieces while ignoring the majority of unimportant information. The self-attention mechanism, as a variant of the attention mechanism, reduces reliance on external information and excels at capturing internal correlations within data or features. Applying the self-attention mechanism in text-primarily involves calculating the mutual influence between words to address the issue of long-range dependencies. Additionally, self-attention is the core idea behind transformers. The core formula for key-value attention is as follows:
自注意力结构[6]：本质上，注意力机制旨在从大量数据中选择少量重要信息，并关注这些重要信息，同时忽略大多数不重要的信息。自注意力机制作为注意力机制的变体，减少了对外部信息的依赖，擅长捕捉数据或特征内部的关联。在文本中应用自注意力机制主要涉及计算词语之间的相互影响，以解决长距离依赖问题。此外，自注意力是变换器背后的核心思想。关键值注意力核心公式如下：

Self-attention allows the model to weigh the importance of different words in a sentence when predicting a particular word. It calculates a weighted sum of the values of all words in the sentence, where the weights are determined by the relevance of each word to the target word.
自注意力机制允许模型在预测特定单词时，根据句子中不同单词的重要性进行加权。它计算句子中所有单词值的加权总和，其中权重由每个单词与目标单词的相关性决定。

The self-attention mechanism consists of three steps: calculating the query, key, and value vectors. The query vector represents the word being attended to, while the key vectors represent all the words in the sentence. The value vectors store the information associated with each word. The attention weights are computed by taking the dot product between the query and key vectors, followed by a softmax operation to obtain a distribution over the words.
自注意力机制包括三个步骤：计算查询、键和值向量。查询向量表示正在关注的单词，而键向量表示句子中的所有单词。值向量存储与每个单词相关的信息。通过计算查询和键向量的点积来计算注意力权重，然后通过 softmax 操作获得单词上的分布。

Multi-Head Attention [6]: Multi-head self-attention extends the self-attention mechanism by performing it multiple times in parallel. Each attention head learns to focus on different aspects of the input, capturing different dependencies and patterns. The outputs of the attention heads are then concatenated and linearly transformed to obtain the final representation. By using multiple attention heads, the model can capture both local and global dependencies, allowing for a more comprehensive understanding of the input sequence. This parallelization also enhances the model’s capacity to capture complex relationships between words. The Multi-head attention can be formulated as follows:
多头注意力[6]：多头自注意力通过并行多次执行自注意力机制来扩展自注意力机制。每个注意力头学习关注输入的不同方面，捕捉不同的依赖关系和模式。然后，将注意力头的输出连接起来并进行线性变换，以获得最终的表示。通过使用多个注意力头，模型可以捕捉局部和全局依赖关系，从而更全面地理解输入序列。这种并行化也增强了模型捕捉词语之间复杂关系的能力。多头注意力可以表示如下：

In this case, "$Concat$" means to concatenate the attention calculation results of each head, "$W^{o}$" is the weight matrix of the output layer, used to linearly transform the concatenated results. This yields the output of multi-head attention. In summary, multi-head attention enhances the model’s ability to represent input sequences by performing parallel attention calculations under different linear transformations, then concatenating and linearly transforming the results. This mechanism plays an important role in the Transformer model, helping to handle long-range dependencies and improve model performance.
在这种情况下，“ $Concat$ ”表示将每个头的注意力计算结果进行拼接，“ $W^{o}$ ”是输出层的权重矩阵，用于对拼接的结果进行线性变换。这产生了多头注意力的输出。总的来说，多头注意力通过在不同线性变换下执行并行注意力计算，然后将结果拼接和线性变换，增强了模型表示输入序列的能力。这种机制在 Transformer 模型中起着重要作用，有助于处理长距离依赖关系并提高模型性能。

The encoder module [6] of the Transformer model is composed of multiple identical layers, each of which includes a multi-head attention mechanism and feed-forward neural network [31]. In the multi-head attention mechanism, each position in the input sequence is calculated for attention with other positions to capture the dependencies between different positions in the input sequence. The feed-forward neural network is then used to further process and extract features from the output of the attention mechanism. The encoder module gradually extracts features of the input sequence through the stacking of multiple such layers and passes the final encoding result to the decoder module for decoding. The design of the encoder module enables it to effectively handle long-range dependencies within the input sequence and has significantly improved performance in various NLP tasks.
Transformer 模型的编码器模块[6]由多个相同的层组成，每一层都包含多头注意机制和前馈神经网络[31]。在多头注意机制中，输入序列中的每个位置都会与其他位置进行注意力计算，以捕获输入序列中不同位置之间的依赖关系。然后，使用前馈神经网络进一步处理和从注意力的输出中提取特征。编码器模块通过堆叠多个这样的层逐渐提取输入序列的特征，并将最终的编码结果传递给解码器模块进行解码。编码器模块的设计使其能够有效地处理输入序列中的长距离依赖关系，并在各种 NLP 任务中显著提高了性能。

The decoder module [32] of the Transformer model is also composed of multiple identical layers, each of which includes a multi-head attention mechanism and a feed-forward neural network. Unlike the encoder, the decoder also includes an additional encoder-decoder attention mechanism, used to compute attention on the input sequence during the decoding process. At each position, the decoder can only perform self-attention calculations with the positions before it to ensure that the generation of the sequence does not violate grammar rules. Masks play an important role in the decoder, ensuring that only information before the current time step is focused on when generating the output sequence, and not leaking information from future time steps. Specifically, the decoder’s self-attention mechanism uses masks to prevent the model from accessing future information when generating predictions at each time step, maintaining the causality of the model. This ensures that the output generated by the model depends on the information at the current time step and before, without being influenced by future information.
Transformer 模型的解码器模块[32]也由多个相同的层组成，每一层都包含多头注意机制和前馈神经网络。与编码器不同，解码器还包括一个额外的编码器-解码器注意机制，用于在解码过程中对输入序列进行注意力计算。在每个位置，解码器只能与它之前的所有位置执行自注意力计算，以确保序列生成不违反语法规则。掩码在解码器中起着重要作用，确保在生成输出序列时只关注当前时间步之前的信息，而不泄露未来时间步的信息。具体来说，解码器的自注意力机制使用掩码来防止模型在生成每个时间步的预测时访问未来信息，保持模型的因果性。这确保了模型生成的输出依赖于当前时间步和之前的信息，而不受未来信息的影响。

Position and order are crucial for certain tasks, such as understanding a sentence or a video. Position and order define the grammar of a sentence, they are integral to the semantics of sentences. The Transformer utilizes Multi-Head Self-Attention (MHSA) to avoid the recursive approach of RNN, thus speeding up the training process. Additionally, it can capture long-range dependencies in sentences and handle longer inputs. When each token in a sentence passes through the Transformer’s Encoder/Decoder stack, the model itself lacks any sense of position/order for each token (permutation invariance). Therefore, a method is still needed to incorporate the sequential information of tokens into the model. To enable the model to perceive the input sequence, positional information about the location of each token in the sentence can be added, and this technique is known as positional embedding (PE). which is used in the Transformer model to incorporate the sequential order of tokens into the input representation. Since the Transformer does not have recurrent connections, it lacks the inherent notion of token order present in recurrent neural networks. To address this, positional embedding assigns a unique vector to each token position in the input sequence. These positional embeddings are added to the word embedding before being fed into the model. By including positional information, the model can differentiate between tokens based on their position in the sequence. In the Transformer model, the core formula of the position embedding can be expressed as:
位置和顺序对于某些任务至关重要，例如理解句子或视频。位置和顺序定义了句子的语法，它们是句子语义的组成部分。Transformer 利用多头自注意力（MHSA）来避免 RNN 的递归方法，从而加快训练过程。此外，它还可以捕捉句子中的长距离依赖关系并处理更长的输入。当句子中的每个标记通过 Transformer 的编码器/解码器堆栈时，模型本身对每个标记的位置/顺序没有任何感知（排列不变性）。因此，仍然需要一种方法将标记的顺序信息纳入模型。为了使模型能够感知输入序列，可以添加每个标记在句子中的位置信息，这种技术被称为位置嵌入（PE），它用于 Transformer 模型将标记的顺序顺序纳入输入表示。由于 Transformer 没有循环连接，它缺乏循环神经网络中固有的标记顺序概念。 为了解决这个问题，位置嵌入为输入序列中的每个标记位置分配一个唯一的向量。这些位置嵌入被添加到词嵌入中，然后再输入到模型中。通过包含位置信息，模型可以根据标记在序列中的位置来区分不同的标记。在 Transformer 模型中，位置嵌入的核心公式可以表示为：

In this equation, $PE$ represents the position embedding matrix, $pos$ represents the position of a token in the sentence, $i$ represents the dimension index of the position embedding, and $d_{model}$ represents the hidden layer dimension of the Transformer model. By using sine and cosine functions and performing different calculations on the position (pos) and dimension (i), this formula generates unique position embedding values for each position and dimension. As a result, each token is assigned a unique position embedding vector, allowing the model to perceive the sequential information of tokens in the sentence. In practical applications, the position embedding matrix is added to the input word embedding matrix to combine position information and semantic information, thereby providing a more comprehensive input representation for the Transformer model.
在这个方程中， $PE$ 表示位置嵌入矩阵， $pos$ 表示句子中标记的位置， $i$ 表示位置嵌入的维度索引， $d_{model}$ 表示 Transformer 模型的隐藏层维度。通过使用正弦和余弦函数，并对位置（pos）和维度（i）进行不同的计算，该公式为每个位置和维度生成独特的位置嵌入值。因此，每个标记被分配一个独特的位置嵌入向量，使模型能够感知句子中标记的顺序信息。在实际应用中，将位置嵌入矩阵添加到输入词嵌入矩阵中，以结合位置信息和语义信息，从而为 Transformer 模型提供更全面的输入表示。

Two commonly used positional encoding methods in Transformer are Absolute Positional Encoding and Relative Positional Encoding.
两种在 Transformer 中常用的位置编码方法是绝对位置编码和相对位置编码。

(1) Absolute Positional Encoding: It generates unique positional embedding values for each position and dimension by using sine and cosine functions. This method uses sine and cosine functions in the mentioned formula to calculate the positional embedding values and adds them to the word embeddings. Absolute Positional Encoding provides a unique encoding for each position, enabling the model to perceive the sequential information of words in the sentence.
(1) 绝对位置编码：它通过使用正弦和余弦函数为每个位置和维度生成独特的位置嵌入值。该方法使用正弦和余弦函数在所述公式中计算位置嵌入值，并将它们添加到词嵌入中。绝对位置编码为每个位置提供独特的编码，使模型能够感知句子中词语的顺序信息。

(2) Relative Positional Encoding: It is an encoding method based on relative positional relationships. Relative Positional Encoding represents positional information by calculating the relative distances between words. This method is used in models like Transformer-XL [33], and Relative Positional Encoding can better capture the relative positional relationships between words when dealing with long sequences. Both of these positional encoding methods aim to provide the positional information of words in the input sequence to the Transformer model, enabling the model to better comprehend and process sequential data. The specific choice of positional encoding method depends on the specific application scenario and model design.
(2) 相对位置编码：这是一种基于相对位置关系的编码方法。相对位置编码通过计算单词之间的相对距离来表示位置信息。这种方法在 Transformer-XL [33]等模型中使用，相对位置编码在处理长序列时能更好地捕捉单词之间的相对位置关系。这两种位置编码方法都旨在向 Transformer 模型提供输入序列中单词的位置信息，使模型能够更好地理解和处理序列数据。具体选择哪种位置编码方法取决于特定的应用场景和模型设计。

There are also other positional encoding methods applied to other models, such as RoPE [34] and ALiBi [35].
也存在其他应用于其他模型的定位编码方法，例如 RoPE [34] 和 ALiBi [35]。

RoPE is a method that uses Absolute Positional Encoding to represent Relative Positional Encoding and is applied in the design of large language models like PaLM [36], LLaMA [9], and GLM-130B [37].
RoPE 是一种使用绝对位置编码来表示相对位置编码的方法，并应用于 PaLM [36]、LLaMA [9]和 GLM-130B [37]等大型语言模型的设计中。

ALiBi does not add positional embeddings to word embeddings but instead adds a pre-defined bias matrix to the attention score based on the distance between tokens. It is applied in the design of large language models like BLOOM [38].
ALiBi 不对词嵌入添加位置嵌入，而是根据标记之间的距离向注意力分数添加预定义的偏置矩阵。它应用于 BLOOM [38]等大型语言模型的设计中。

Some other positional encoding methods, such as mixed positional encoding, multi-digit positional encoding, and implicit positional encoding, are also used by some models.
一些其他的位置编码方法，如混合位置编码、多数字位置编码和隐式位置编码，也被某些模型所使用。

Prompt learning is a new approach to machine learning [51]. In the early field of natural language processing (NLP), researchers mainly used fully supervised learning mode[52], which trained models for specific tasks on the input and output example dataset of the target task. However, due to the limited training dataset, this method cannot train high-quality models well, so early NLP relied more on feature engineering; With the emergence of neural network models and their use in the field of NLP, people have begun to pay attention to architecture engineering [53].
提示学习是机器学习的一种新方法[ 51]。在自然语言处理（NLP）的早期领域，研究人员主要使用全监督学习模式[ 52]，在目标任务的输入和输出示例数据集上训练特定任务的模型。然而，由于训练数据集有限，这种方法无法很好地训练高质量的模型，因此早期的 NLP 更多地依赖于特征工程；随着神经网络模型的出现及其在 NLP 领域的应用，人们开始关注架构工程[ 53]。

However, between 2017 and 2019, the learning approach of NLP models shifted from fully supervised learning to a new mode: pre-train and fine-tune paradigm[54]. In this paradigm, a model with a fixed architecture is pre-trained as a language model to predict the probability of observed text data. Due to the abundant raw text data required for training language models, these language models can be trained on large datasets. During this process, language models can learn robust universal features of the language they are modeling. Then, by introducing additional parameters and fine-tuning them using task-specific objective functions, the PLM mentioned above will adapt to different downstream tasks. At this point, the focus of research shifted to objective engineering, which is to design training objectives during pre-training and fine-tuning. Since BERT, NLP has been using pre-training and fine-tuning methods for a long period of time, but this approach requires a new model to be fine-tuned for each task and cannot be shared. But for an LLM, it feels like customizing each task, which is very inefficient [51].
然而，在 2017 年至 2019 年之间，NLP 模型的学习方法从完全监督学习转变为一种新模式：预训练和微调范式[54]。在这种范式下，一个具有固定架构的模型被预训练为语言模型，以预测观察到的文本数据的概率。由于训练语言模型需要大量的原始文本数据，这些语言模型可以在大型数据集上训练。在这个过程中，语言模型可以学习到它们所建模的语言的鲁棒通用特征。然后，通过引入额外的参数并使用特定任务的客观函数来微调这些参数，上述 PLM 将适应不同的下游任务。此时，研究重点转向了目标工程，即在预训练和微调期间设计训练目标。自从 BERT 以来，NLP 已经长期使用预训练和微调方法，但这种方法需要为每个任务微调一个新的模型，并且不能共享。但对于一个LLM来说，感觉像是为每个任务定制，这非常低效[51]。

Prompt learning, this method has demonstrated amazing capabilities in GPT-3. The GPT-3 model can handle many tasks with only a few samples by using natural language prompts and task demonstrations as context, without updating parameters in the underlying model. Prompt Learning replaces the process of pre-trained and fine-tuning with pre-trained, prompts and predictions. In this paradigm, the downstream task is not to adapt the pre-trained LM to the downstream task through objective engineering, but to redefine the downstream task with the help of text prompts, making it look more like the tasks solved during the original LM training. For prompt learning, it is only necessary to insert different prompt parameters to adapt to different tasks. That is to say, each task only needs to train the prompt parameter separately, without the need to train the entire pre-trained language model[55]. This approach greatly improves the efficiency of using pre-trained language models and significantly shortens training time.
提示学习，这种方法在 GPT-3 中展示了惊人的能力。GPT-3 模型可以通过使用自然语言提示和任务演示作为上下文，仅用少量样本就能处理许多任务，而无需更新底层模型的参数。提示学习用预训练、提示和预测取代了预训练和微调的过程。在这个范式下，下游任务不是通过目标工程将预训练的 LM 适应于下游任务，而是通过文本提示重新定义下游任务，使其更像在原始 LM 训练期间解决的那些任务。对于提示学习，只需插入不同的提示参数来适应不同的任务即可。也就是说，每个任务只需要单独训练提示参数，无需训练整个预训练语言模型[55]。这种方法大大提高了使用预训练语言模型的效率，并显著缩短了训练时间。

In the traditional pre-trained+fine-tuning paradigm, there is a gap between the pre-trained stage and downstream tasks [51], while prompt learning can maintain consistency between the pre-trained target format and downstream task output format, that is, align the form of downstream tasks with the form of PLMs pre-trained tasks. When training PLMs, we can transform the original target task into a fill-in-the-blank or continuation task similar to the pre-trained task of PLMs by constructing a prompt. The advantage of this method is that through a series of appropriate prompts, we can use a single language model to solve various downstream tasks.
在传统的预训练+微调范式下，预训练阶段与下游任务之间存在差距[51]，而提示学习可以保持预训练目标格式与下游任务输出格式的一致性，即对齐下游任务的形式与 PLMs 预训练任务的形式。在训练 PLMs 时，我们可以通过构建提示，将原始目标任务转换为类似于 PLMs 预训练任务的填空或续写任务。这种方法的优势在于，通过一系列适当的提示，我们可以使用单个语言模型来解决各种下游任务。

Prompt learning optimizes the performance of models on different tasks by using pre-trained models and designing appropriate templates. Prompt learning consists of prompt templates, answer mappings, and pre-trained language models. The prompt template is the main body of the prompt, and fill in the blank [56] and generate based on prefix [57]are two common types of prompt learning templates. The fill-in-the-blank template selects one or more positions in the text and represents them with [MASK] tags, used to prompt the model to fill in the corresponding words; Prefix-based template generation involves adding a specific prefix before a sentence to guide the model in generating appropriate text. Answer mapping is the process of evaluating all possible answers according to a probability distribution, selecting the most likely answer as the predicted output, and converting it into appropriate category mapping words. This process typically involves converting labels into natural language vocabulary, known as Verbalizer [58].
提示学习通过使用预训练模型和设计适当的模板来优化不同任务上模型的性能。提示学习包括提示模板、答案映射和预训练语言模型。提示模板是提示的主要内容，填空[56]和基于前缀[57]生成是两种常见的提示学习模板。填空模板在文本中选择一个或多个位置并用[56]标签表示，用于提示模型填写相应的词语；基于前缀的模板生成涉及在句子前添加特定的前缀，以引导模型生成适当的文本。答案映射是根据概率分布评估所有可能的答案，选择最可能的答案作为预测输出，并将其转换为适当的分类映射词语的过程。此过程通常涉及将标签转换为自然语言词汇，称为 Verbalizer[58]。

The workflow of Prompt learning mainly includes the following four parts:
工作流学习的主要流程主要包括以下四个部分：

(1)Use PLMs as base encoders
(1)使用 PLM 作为基础编码器

(2)Add additional context (template) with a [MASK] position
(2)添加额外的上下文（模板）与一个位置

(3)Project labels to label words (verbalizer)
(3)将项目标签映射到标签词（词化器）

(4)Be the GAP between pre-training and fine-tuning
（4）成为预训练和微调之间的差距

After defining the template and answer space, we need to choose a suitable pre-trained language model. There are now various pre-trained models (PTMs) with good performance, and when selecting a model, one usually considers its paradigm, such as Auto recursive, Masked Language Modeling, Encoder Decoder, etc. Based on this, for the summary task, a more suitable Bidirectional and Auto-Regressive Transformers (BART) model can be selected.
在定义模板和答案空间后，我们需要选择一个合适的预训练语言模型。现在有各种性能良好的预训练模型（PTMs），在选择模型时，人们通常考虑其范式，例如自动递归、掩码语言建模、编码器-解码器等。基于此，对于摘要任务，可以选择更合适的双向和自回归变换器（BART）模型。

The selection of a template plays a very important role in the prompt learning. Templates can generally be distinguished based on whether they are manually specified: artificially constructed templates or automatically searched templates. Artificially created templates are the most intuitive method, easy to understand, and have good performance in practical applications. However, artificially constructed templates also have some drawbacks: prior knowledge is required when designing templates manually [59], and there may be failures [60]. There are two types of automatically generated templates: discrete prompts and continuous prompts. Discrete prompts allow the model to select the optimal template in a set of discrete template spaces, while continuous prompts allow the language model to automatically train a prompt. According to research, using multiple templates [61] can improve the performance of the model. The simplest way to choose to use multiple templates and aggregate them together to complete an answer is to take the average [60] or weighted average of each template output [58].
模板选择在提示学习中起着非常重要的作用。模板通常可以根据是否手动指定来区分：人工构建的模板或自动搜索的模板。人工创建的模板是最直观的方法，易于理解，在实际应用中表现良好。然而，人工构建的模板也有一些缺点：设计模板时需要先验知识[59]，并且可能存在失败[60]。自动生成的模板有两种类型：离散提示和连续提示。离散提示允许模型在离散模板空间中选择最优模板，而连续提示允许语言模型自动训练提示。根据研究，使用多个模板[61]可以提高模型性能。选择使用多个模板并将其汇总以完成答案的最简单方法是取平均值[60]或每个模板输出的加权平均值[58]。

Verbalizer is the process of mapping labels to label words, and the selection of verbalizers is also crucial for prompt learning. There are two ways to construct a verbalizer: manual definition and automatic search. The manual definition requires professional knowledge and may have disadvantages such as strong subjectivity and a small coverage area. To solve this problem, we can choose the following solutions: (1) Manually design with human prior knowledge; (2) Start with an Intel label word, paraphrase and expand; (3) Start with an internal label word, using external knowledge and expand; (4) Decompose the label with multiple tokens; (5) Virtual token and optimize the label embedding. In addition, we can use external knowledge bases to expand and improve label words, thereby achieving better text classification results[62].
Verbalizer 是将标签映射到标签词的过程，选择 Verbalizer 对于提示学习也非常关键。构建 Verbalizer 有两种方式：手动定义和自动搜索。手动定义需要专业知识，可能存在主观性强、覆盖面小等缺点。为解决这个问题，我们可以选择以下方案：（1）手动设计，利用人类先验知识；（2）从 Intel 标签词开始，进行释义和扩展；（3）从内部标签词开始，利用外部知识进行扩展；（4）用多个标记分解标签；（5）虚拟标记和优化标签嵌入。此外，我们可以使用外部知识库来扩展和改进标签词，从而实现更好的文本分类结果[62]。

The emergence of the new paradigm of Prompt learning has brought significant changes to the training process. The learning strategies for Prompt learning mainly include the following: (1) Pre-training then fine-tuning, which is a traditional pre-training+fine tuning method [63]; (2) Tuning free promotion, relying on the designer LM of prompts to directly provide answers [64]; (3) Fixed LM prompt tuning, which updates the relevant parameters of prompts using downstream task training data; (4) Fix prompt LM tuning, this strategy is to fine-tune the parameters of LM, which have fixed parameters when using prompts; (5) Prompt+LM tuning is a strategy that updates both prompts related parameters and LM parameters.
新范式 Prompt 学习的出现给训练过程带来了重大变化。Prompt 学习的策略主要包括以下几种：(1)先预训练后微调，这是一种传统的预训练+微调方法[63]；(2)无调优自由推广，依赖于提示的设计者 LM 直接提供答案[64]；(3)固定 LM 提示微调，使用下游任务训练数据更新提示的相关参数；(4)固定提示 LM 微调，这种策略是微调具有固定参数的 LM 的参数；(5)Prompt+LM 微调是一种更新与提示相关参数和 LM 参数的策略。

These different learning strategies can be selected based on specific tasks and needs. Pre-training + fine-tuning is the most common strategy, suitable for most tasks [63]. No fine-tuning prompts are suitable for simple tasks, which can greatly reduce training time and computational resource consumption. Fixed LM prompt fine-tuning and fixed prompt LM fine-tuning are suitable for tasks that require more precise control and can optimize model performance by adjusting prompt parameters or language model parameters. Combining prompts and LM fine-tuning combines the advantages of both and can further improve model performance [51].
这些不同的学习策略可以根据具体任务和需求进行选择。预训练+微调是最常见的策略，适用于大多数任务[63]。没有微调提示适用于简单任务，这可以大大减少训练时间和计算资源消耗。固定 LM 提示微调和固定提示 LM 微调适用于需要更精确控制的任务，可以通过调整提示参数或语言模型参数来优化模型性能。将提示和 LM 微调相结合结合了两者的优点，可以进一步提高模型性能[51]。

In summary, Prompt learning provides us with a new training paradigm that can optimize model performance on various downstream tasks through appropriate prompt design and learning strategies. Choosing the appropriate template, constructing an effective verbalizer, and adopting appropriate learning strategies are all important factors in improving the effectiveness of prompt learning.
总结来说，提示学习为我们提供了一种新的训练范式，通过适当的提示设计和学习策略，可以在各种下游任务中优化模型性能。选择合适的模板、构建有效的表述器和采用适当的学习策略都是提高提示学习有效性的重要因素。

Training LLMs require vast amounts of text data, and the quality of this data significantly impacts LLM performance. Pre-training on large-scale corpora provides LLMs with a fundamental understanding of language and some generative capability. The first step in LLM training is collecting substantial corpora of natural language text. Pre-training data sources are diverse, commonly incorporating web text, conversational data, and books as general pre-training corpora. Additionally, some research efforts introduce specialized data from professional domains, such as code or scientific data, to enhance LLM capabilities in those fields. Leveraging diverse sources of text data for LLM training can significantly enhance the model’s generalization capabilities. In the following section, we will present the commonly used datasets for training LLMs as shown in Table 1. These corpora are categorized into 5 groups for discussion.
训练需要大量文本数据，这些数据的质量对性能有显著影响。在大规模语料库上进行预训练为提供了对语言的基本理解和一些生成能力。训练的第一步是收集大量的自然语言文本语料库。预训练数据来源多样，通常包括网络文本、对话数据和书籍作为一般预训练语料库。此外，一些研究工作引入了来自专业领域的专业数据，如代码或科学数据，以增强在这些领域的功能。利用多样化的文本数据来源进行训练可以显著提高模型的一般化能力。在下一节中，我们将展示如表 1 所示的常用训练数据集。这些语料库被分为 5 组进行讨论。

Books: Two commonly utilized books datasets for LLMs training are BookCorpus [65] and Gutenberg [66]. These datasets include a wide range of literary genres, including novels, essays, poetry, history, science, philosophy, and more. Widely employed by numerous LLMs [9; 79], these datasets contribute to the models’ training by exposing them to a diverse array of textual genres and subject matter, fostering a more comprehensive understanding of language across various domains.
书籍：用于LLMs训练的两种常用书籍数据集是 BookCorpus [65]和 Gutenberg [66]。这些数据集包括广泛的文学体裁，包括小说、散文、诗歌、历史、科学、哲学等。被众多LLMs [9; 79]广泛使用，这些数据集通过使模型接触多样化的文本体裁和主题，促进了模型对语言在各个领域的更全面理解。

CommonCrawl: CommonCrawl [67] manages an accessible repository of web crawl data, freely available for utilization by individuals and organizations. This repository encompasses a vast collection of data, comprising over 250 billion web pages accumulated over a span of 16 years. Established in 2007, Common Crawl has evolved into a widely recognized and referenced corpus in the academic and research communities, cited in more than 10,000 research papers. This continuously expanding corpus is a dynamic resource, with an addition of 3–5 billion new web pages each month. Its significance extends to the field of natural language processing, where it serves as a primary training corpus in numerous large language models. Notably, a substantial portion of the raw tokens employed in training GPT-3 [8], amounting to 82%, is sourced from the CommonCrawl. However, due to the presence of a substantial amount of low-quality data in web archives, preprocessing is essential when working with CommonCrawl data. Currently, four commonly used filtered datasets based on CommonCrawl are available: C4 [68], CC-Stories [69], CC-News [70], and RealNews [71].
CommonCrawl：CommonCrawl [67] 管理着一个可访问的网页爬虫数据仓库，个人和组织可以免费使用。该仓库包含大量数据，包括 16 年间积累的超过 2500 亿个网页。成立于 2007 年，Common Crawl 已成为学术界和研究界广泛认可和引用的语料库，被超过 10,000 篇研究论文引用。这个持续扩展的语料库是一个动态资源，每月新增 3-5 亿个新网页。它在自然语言处理领域具有重要意义，是许多大型语言模型的主要训练语料库。值得注意的是，用于训练 GPT-3 [8] 的原始标记中有相当一部分，即 82%，来自 CommonCrawl。然而，由于网络档案馆中存在大量低质量数据，处理 CommonCrawl 数据时进行预处理是必要的。目前，基于 CommonCrawl 的四个常用过滤数据集可供使用：C4 [68]、CC-Stories [69]、CC-News [70] 和 RealNews [71]。

Reddit Links: Reddit is a social media platform where users can submit links and posts, and others can vote on them using the "upvote" or "downvote" system. This characteristic makes it a valuable resource for creating high-quality datasets.
Reddit 链接：Reddit 是一个社交媒体平台，用户可以提交链接和帖子，其他人可以使用“点赞”或“踩”系统对它们进行投票。这一特性使其成为创建高质量数据集的有价值资源。

Wikipedia: Wikipedia [75], a free and open online encyclopedia project, hosts a vast repository of high-quality encyclopedic content spanning a wide array of topics. The English version of Wikipedia is extensively utilized in the training of many LLMs [8; 9; 80], serving as a valuable resource for language understanding and generation tasks. Additionally, Wikipedia is available in multiple languages, providing diverse language versions that can be leveraged for training in multilingual environments.
维基百科：维基百科[75]，一个免费和开放的在线百科全书项目，拥有涵盖广泛主题的高质量百科内容库。维基百科的英文版本被广泛用于许多LLMs [8；9；80]的训练中，作为语言理解和生成任务的有价值资源。此外，维基百科提供多种语言版本，可在多语言环境中用于训练。

Code: There is a limited availability of publicly accessible code datasets at present. Existing efforts primarily involve web scraping of code with open-source licenses from the internet. The main sources include Github and Stack Overflow.
目前公开可访问的代码数据集数量有限。现有的努力主要涉及从互联网上开源许可的代码中进行网络爬取。主要来源包括 GitHub 和 Stack Overflow。

We have organized datasets utilized by distinct LLMs. During the training process, LLMs are typically trained on multiple datasets, as specified in Table 2 for reference.
我们已组织了由不同 LLMs 使用的数据集。在训练过程中，LLMs 通常在多个数据集上进行训练，如表 2 所示供参考。

Once an adequate corpus of data is collected, the subsequent step is data preprocessing. The quality of data preprocessing directly impacts the model’s performance and security. The specific preprocessing steps involve filtering low-quality text, including eliminating toxic and biased content to ensure the model aligns with human ethical standards. It also includes deduplication, removing duplicates in the training set, and excluding redundant content in the test set to maintain the sample distribution balance. Privacy scrubbing is applied to ensure the model’s security, preventing information leakage or other privacy-related concerns. Additionally, if fine-tuning LLMs is considered, expanding the vocabulary should also be considered. On the other hand, LLaMA 2 models [10] represent a notable exception. These models forego filtering in their pretraining corpus, as aggressive filtration might accidentally filter out some demographic groups. This approach enhances the generalizability of the base LLaMA 2 models, making them more adept across a range of downstream tasks, such as hate speech detection and privacy de-identification. Observations indicate that abstaining from additional filtering in the pretraining data enables the base model to achieve reasonable safety alignment with fewer examples [10]. While this increases both generalizability and safety alignment efficiency, the implementation of additional safety mitigations is still imperative prior to public deployment, as further discussed in Section 3.5.4.
一旦收集到足够的数据语料库，下一步就是数据预处理。数据预处理的质最直接影响模型的性能和安全。具体的预处理步骤包括过滤低质量文本，包括消除有毒和偏见内容，以确保模型符合人类的伦理标准。这还包括去重，在训练集中删除重复项，以及在测试集中排除冗余内容，以保持样本分布平衡。应用隐私擦除以确保模型的安全，防止信息泄露或其他隐私相关的问题。此外，如果考虑在LLMs上进行微调，也应考虑扩展词汇。另一方面，LLaMA 2 模型[10]是一个值得注意的例外。这些模型在预训练语料库中放弃了过滤，因为激进的过滤可能会意外地过滤掉某些人口群体。这种方法增强了基础 LLaMA 2 模型的可推广性，使它们在一系列下游任务中更加熟练，例如仇恨言论检测和隐私去识别。 观察表明，在预训练数据中避免额外的过滤可以使基础模型在更少的示例中实现合理的安全性对齐[10]。虽然这增加了泛化能力和安全性对齐效率，但在公共部署之前实施额外的安全缓解措施仍然至关重要，如第 3.5.4 节所述。

Quality filtering: Filtering low-quality data is typically done using heuristic-based methods or classifier-based methods. Heuristic methods involve employing manually defined rules to eliminate low-quality data [84; 72]. For instance, rules could be set to retain only text containing digits, discard sentences composed entirely of uppercase letters, and remove files with a symbol and word ratio exceeding 0.1, and so forth. Classifier-based methods involve training a classifier on a high-quality dataset such as WebText [85] to filter out low-quality datasets.
质量过滤：通常使用基于启发式的方法或基于分类器的方法来过滤低质量数据。启发式方法涉及使用手动定义的规则来消除低质量数据[84; 72]。例如，可以设置规则仅保留包含数字的文本，丢弃完全由大写字母组成的句子，以及移除符号和词比例超过 0.1 的文件等。基于分类器的方法涉及在高质量数据集（如 WebText[85]）上训练一个分类器，以过滤出低质量数据集。

Deduplication: Language models may sometimes repetitively generate the same content during text generation, potentially due to a high degree of repetition in the training data. Extensive repetition can lead to training instability, resulting in a decline in the performance of LLMs [86]. Additionally, it is crucial to consider avoiding dataset contamination by removing duplicated data present in both the training and testing set [87].
去重：语言模型在文本生成过程中有时可能会重复生成相同的内容，这可能是由于训练数据中重复程度较高所致。大量的重复可能导致训练不稳定，从而降低LLMs [ 86]的性能。此外，考虑避免数据集污染，通过移除训练集和测试集中存在的重复数据至关重要[ 87]。

Privacy scrubbing: LLMs, as text-generating models, are trained on diverse datasets, which may pose privacy concerns and the risk of inadvertent information disclosure [88]. In the preprocessing phase of language datasets, it is imperative to address privacy concerns by systematically removing any sensitive information. This involves employing techniques such as anonymization, redaction, or tokenization to eliminate personally identifiable details, geolocation, and other confidential data. By carefully scrubbing the dataset of such sensitive content, researchers and developers can ensure that the language models trained on these datasets uphold privacy standards and mitigate the risk of unintentional disclosure of private information. It is essential to strike a balance between data utility and privacy protection, fostering responsible and ethical use of language datasets in various applications.
隐私清理：LLMs作为文本生成模型，在多样化的数据集上进行训练，可能会引发隐私担忧和意外信息泄露的风险[88]。在语言数据集的前处理阶段，系统地删除任何敏感信息以解决隐私问题至关重要。这包括采用匿名化、编辑或标记化等技术来消除个人身份信息、地理位置和其他机密数据。通过仔细清理数据集中此类敏感内容，研究人员和开发者可以确保在这些数据集上训练的语言模型遵守隐私标准并降低无意中泄露私人信息的风险。在数据实用性和隐私保护之间取得平衡，对于在各个应用中负责任和道德地使用语言数据集至关重要。

Filtering out toxic and biased text: In the preprocessing steps of language datasets, a critical consideration is the removal of toxic and biased content to ensure the development of fair and unbiased language models. This involves implementing robust content moderation techniques, such as employing sentiment analysis, hate speech detection, and bias identification algorithms. By leveraging these tools [89], researchers can systematically identify and filter out text that may perpetuate harmful stereotypes, offensive language, or biased viewpoints.
过滤掉有毒和偏见文本：在语言数据集的前处理步骤中，一个关键考虑因素是移除有毒和偏见内容，以确保公平和无偏见的语言模型的发展。这涉及到实施强大的内容审查技术，例如采用情感分析、仇恨言论检测和偏见识别算法。通过利用这些工具[89]，研究人员可以系统地识别和过滤掉可能 perpetuate harmful stereotypes, offensive language, or biased viewpoints 的文本。

The Encoder-decoder architecture of LLMs is built upon the traditional Transformer Encoder-decoder architecture. The Encoder-decoder architecture consists of two main components: the Encoder and the Decoder. Each part of the Encoder is composed of multiple layers of Transformer’s Multi-Head Self-Attention layers, which encode the input sequence. The Decoder, on the other hand, utilizes cross-attention over the output representation of the Encoder and generates the target sequence in an autoregressive manner. The encoder-decoder architecture serves as the foundation for prominent LLMs such as T5 [68], flan-T5 [91], and BART [92].
LLMs的编码器-解码器架构建立在传统的 Transformer 编码器-解码器架构之上。编码器-解码器架构由两个主要组件组成：编码器和解码器。编码器的每一部分由多个 Transformer 的多头自注意力层组成，用于编码输入序列。另一方面，解码器利用对编码器输出表示的交叉注意力，以自回归的方式生成目标序列。编码器-解码器架构是 T5 [68]、flan-T5 [91]和 BART [92]等突出架构的基础。

LLMs with a Decoder-only architecture utilize the decoder component of the traditional Transformer architecture. Unlike the Encoder-Decoder architecture, which incorporates both an encoder and a decoder, the Decoder-only architecture is solely focused on the decoding process. In this configuration, the model sequentially generates tokens, attending to preceding tokens in the sequence. This architecture has been applied to various language generation tasks, showcasing its effectiveness in various tasks such as text generation without the need for an explicit encoding phase. The Decoder-only architecture can be further classified into two categories: the Causal Decoder architecture and the Prefix Decoder architecture.
仅使用解码器架构利用传统 Transformer 架构的解码器组件。与包含编码器和解码器的编码器-解码器架构不同，仅解码器架构专注于解码过程。在这种配置中，模型按顺序生成标记，关注序列中的先前标记。这种架构已应用于各种语言生成任务，展示了其在各种任务（如无需显式编码阶段进行文本生成）中的有效性。仅解码器架构可以进一步分为两类：因果解码器架构和前缀解码器架构。

The Causal Decoder Architecture: In the Causal Decoder architecture, each token in the model input sequence can only attend to past input tokens and itself during the decoding process. It achieves unidirectional attention to the input sequence by using a specific mask as shown in Figure 1. In fact, different architectures are mainly implemented by configuring different mask matrices. The figure illustrates a comparison of mask configurations between the Encoder-decoder and Decoder-only architectures (including Casual Decoder and Prefix Decoder). The representative LLMs for the Causal Decoder architecture are the GPT series [18; 7; 8; 93; 19]. The GPT series of LLMs are currently known for their superior performance, with their foundational Causal Decoder architecture widely applied in other LLMs such as BLOOM [38], OPT [83], Gopher [84], and LLaMA [9].
因果解码器架构：在因果解码器架构中，模型输入序列中的每个标记在解码过程中只能关注过去的输入标记和自身。它通过使用如图 1 所示的特定掩码实现了对输入序列的单向注意力。实际上，不同的架构主要是通过配置不同的掩码矩阵来实现的。该图展示了编码器-解码器架构和仅解码器架构（包括因果解码器和前缀解码器）之间的掩码配置比较。因果解码器架构的代表性LLMs是 GPT 系列[18; 7; 8; 93; 19]。LLMs的 GPT 系列目前以其卓越的性能而闻名，其基础因果解码器架构广泛应用于其他LLMs，如 BLOOM[38]、OPT[83]、Gopher[84]和 LLaMA[9]。

The Prefix Decoder Architecture: The Prefix Decoder architecture combines the advantages of both the Encoder-decoder and Causal Decoder architectures. It leverages its unique mask configurations, as illustrated in Figure 1, enabling bidirectional attention for tokens in the prefix while maintaining unidirectional attention for generating subsequent tokens [54]. This design allows for the autoregressive generation of the output sequence with the flexibility to attend bi-directionally to the prefix tokens. Representative LLMs utilizing the Prefix Decoder architecture include PaLM [36] and GLM [37].
前缀解码器架构：前缀解码器架构结合了编码器-解码器和因果解码器架构的优点。它利用其独特的掩码配置，如图 1 所示，使前缀中的标记能够进行双向注意力，同时在生成后续标记时保持单向注意力[54]。这种设计允许以自回归的方式生成输出序列，同时具有对前缀标记进行双向注意力的灵活性。采用前缀解码器架构的代表性模型包括 PaLM[36]和 GLM[37]。

In the parallel training mentioned below, there will be discussions about collective communication which helps us better understand the principles of parallel training. Figure 2 has listed five reduction relationships. 1)Broadcast: Send data from one GPU to other GPUs.2)Reduce: Reduce(sum/average) data of all GPUs, send to one GPU.3)All Reduce: Reduce all data of GPUs, send to all GPUs.4)Reduce Scatter: Reduce all data of GPUs, send portions to all GPUs.5)All Gather: Gather data of all GPUs, send all GPUs.
在以下并行训练中，将讨论集体通信，这有助于我们更好地理解并行训练的原则。图 2 列出了五种缩减关系。1）广播：将数据从一块 GPU 发送到其他 GPU。2）缩减（求和/平均）：缩减所有 GPU 的数据，发送到一块 GPU。3）全缩减：缩减所有 GPU 的所有数据，发送到所有 GPU。4）缩减散射：缩减所有 GPU 的所有数据，将部分数据发送到所有 GPU。5）全收集：收集所有 GPU 的数据，发送到所有 GPU。

Data Parallel: The process of data parallelism [94] is shown in Figure 3, there is a parameter server that stores the model’s parameters and the entire batch of data. Each GPU uses broadcast to synchronize the model parameters and divides the data into one portion per GPU, with each GPU receiving a portion of the data. Each GPU uses the complete model parameters and a portion of the data to perform forward and backward propagation. This way, the gradients are obtained for each GPU. Finally, we aggregate the gradients and send the aggregated gradients back to the parameter server, where the original model parameters and the aggregated complete gradients are available. With this information, we can use an optimizer to update the model parameters. The updated parameters will then enter the next round of model training iterations.
数据并行：数据并行化过程[94]如图 3 所示，有一个参数服务器存储模型的参数和整个数据批次。每个 GPU 使用广播来同步模型参数，并将数据分成每个 GPU 一部分，每个 GPU 接收数据的一部分。每个 GPU 使用完整的模型参数和数据的一部分进行前向和反向传播。这样，为每个 GPU 获得梯度。最后，我们汇总梯度并将汇总的梯度发送回参数服务器，在那里可以找到原始模型参数和汇总的完整梯度。有了这些信息，我们可以使用优化器来更新模型参数。更新后的参数将进入下一轮模型训练迭代。

Distributed data parallelism [95] abandons the use of a parameter server and instead employs all-reduce on gradient information, ensuring that each GPU receives the same gradient information. The result of all-reduce is communicated to all GPUs, allowing them to independently update their respective model optimizers. After each round of updates, the model’s parameters, gradients, and the historical information of the optimizer are consistent across all GPUs.
分布式数据并行[95]放弃使用参数服务器，转而采用梯度信息上的 all-reduce，确保每个 GPU 接收相同的梯度信息。all-reduce 的结果被传达给所有 GPU，使它们能够独立更新各自的模型优化器。在每轮更新之后，模型的参数、梯度和优化器的历史信息在所有 GPU 上保持一致。

The occupation of GPU memory of intermediate results is related to the batch size, sentence length, and model dimensions. When using data parallelism, a batch of data is divided into many parts, allowing each GPU to process a portion of the data. In equivalent terms, the batch size processed on each GPU is reduced to one over the original number of GPUs. Data parallelism has reduced the input dimensions, resulting in an overall reduction in the intermediate results of the model. A drawback is that to support model training, each GPU needs to receive at least one piece of data. In the most extreme case, when each GPU receives only one piece of data, our parameters, gradients, and optimizer still need to be fully stored on the GPU. Even if we don’t store any intermediate results on the GPU, our model may still be unable to perform computations on a single GPU.
GPU 内存被中间结果占用与批量大小、句子长度和模型维度有关。在使用数据并行时，一个数据批次被分成多个部分，允许每个 GPU 处理数据的一部分。从等效的角度来看，每个 GPU 处理的批量大小减少到原始 GPU 数量的倒数。数据并行减少了输入维度，导致模型中间结果的整体减少。缺点是，为了支持模型训练，每个 GPU 至少需要接收一块数据。在最极端的情况下，当每个 GPU 只接收一块数据时，我们的参数、梯度和优化器仍然需要完全存储在 GPU 上。即使我们不存储任何中间结果在 GPU 上，我们的模型可能仍然无法在单个 GPU 上执行计算。

Model Parallel:Model parallelism [96] was first introduced by Megatron-LM to alleviate memory pressure. From figure 4, we can clearly understand the overall architecture of model parallelism. Taking advantage of the most common linear layer in the Transformer as an example, the parameters of the linear layer form a matrix of size A*B, and the input to the linear layer is a vector of size B*1. Representing this as $y_{A*B}$ = $W_{A*B}$$x_{B}$, we can horizontally partition the model’s parameters into many segments using the property of matrix multiplication. Each segment is of size a divided by n multiplied by B. Utilizing the properties of matrix multiplication, we can move $x_{B}$ into parentheses, and finally, the result of the linear layer is obtained by multiplying many small matrices with the parameters of the linear layer. Through this approach, the parameters of the linear layer can be distributed across multiple GPUs. However, it is crucial to ensure that the inputs to the model on multiple GPUs are identical. Instead of using a data parallel approach to partition the data, we need to ensure that the inputs obtained on each GPU are the same, meaning they belong to the same batch of data. We can then partition a parameter like the linear layer across GPUs, with each GPU receiving a small portion of the matrix. By performing model calculations with this small portion and the data, we obtain a sub-result, as shown in Formula 5. The results of these computations need to be concatenated using the all-gather operator and communicated to all GPUs.
模型并行：模型并行性[96]最初由 Megatron-LM 提出，以减轻内存压力。从图 4 中，我们可以清楚地了解模型并行的整体架构。以 Transformer 中最常见的线性层为例，线性层的参数形成一个 A*B 大小的矩阵，线性层的输入是一个 B*1 大小的向量。将此表示为 $y_{A*B}$ = $W_{A*B}$ $x_{B}$ ，我们可以利用矩阵乘法的性质，将模型的参数水平分割成多个段。每个段的大小是 a 除以 n 再乘以 B。利用矩阵乘法的性质，我们可以将 $x_{B}$ 移动到括号内，最终通过乘以多个小矩阵和线性层的参数，得到线性层的输出。通过这种方法，线性层的参数可以分布到多个 GPU 上。然而，确保多个 GPU 上模型的输入相同至关重要。我们不是使用数据并行方法来分割数据，而是需要确保每个 GPU 上获得的输入相同，即它们属于同一批数据。 我们可以将线性层等参数在 GPU 之间进行分区，每个 GPU 接收矩阵的一小部分。通过使用这一小部分数据和模型进行计算，我们得到一个子结果，如公式 5 所示。这些计算的结果需要使用 all-gather 算子进行拼接，并传递给所有 GPU。

ZeRO: ZeRO [97] is a framework built on data parallelism. During the parameter updating process on each GPU, the same set of parameters is used, leading to computational redundancy. Each GPU uses reduced scatter to eliminate this redundancy to obtain a portion of the gradient results. After updating a portion of the model parameters on each GPU, an all-gather operation is performed to synchronize the parameters across all GPUs. After the all-gather operation, the original gradient no longer needs to be saved on the graphics card and can be removed. Figure 5 shows the update of ZeRO. In ZeRO1, the original gradient is removed after backward propagation, while in ZeRO2, the product of the gradient* is calculated in advance during backward propagation, and only the gradient* is saved on the graphics card, removing the gradient. This way, the deletion of the gradient is advanced, leading to further savings in GPU memory space. ZeRO3 conducts a detailed division of the model parameters. Each graphics card retains only a portion of the gradients for updating, and parameter updates also only affect a portion of the model parameters. Therefore, each graphics card only needs to store the parameters, gradients, and optimizer related to the part of the parameters it is responsible for. During forward and backward propagation, an all-gather operation is required once, and after the operation is complete, the model parameters are released from the graphics card.Zero3 does not use all gather during parameter updates, but it requires an all-gather operation during both forward and backward propagation, adding one communication step. Compared to ZeRO2, ZeRO3 is an algorithm that trades time for space.
ZeRO：ZeRO [97] 是一个基于数据并行的框架。在每块 GPU 上的参数更新过程中，使用相同的参数集，导致计算冗余。每个 GPU 使用减少的散射来消除这种冗余，以获得部分梯度结果。在每块 GPU 上更新部分模型参数后，执行全收集操作以同步所有 GPU 上的参数。全收集操作后，原始梯度不再需要在图形卡上保存，可以删除。图 5 显示了 ZeRO 的更新。在 ZeRO1 中，在反向传播后删除原始梯度，而在 ZeRO2 中，在反向传播期间预先计算梯度*的乘积，只保存图形卡上的梯度*，删除梯度。这样，梯度删除被提前，导致 GPU 内存空间的进一步节省。ZeRO3 对模型参数进行了详细划分。每个图形卡仅保留部分梯度以供更新，参数更新也仅影响模型参数的一部分。 因此，每张显卡只需存储其负责参数的部分的参数、梯度和优化器。在正向和反向传播过程中，需要执行一次全聚合操作，操作完成后，模型参数从显卡中释放。Zero3 在参数更新期间不使用全聚合，但在正向和反向传播期间都需要执行全聚合操作，增加了一个通信步骤。与 ZeRO2 相比，ZeRO3 是一种以空间换取时间的算法。

Pipeline Parallel: Pipeline parallelism [98] and model parallelism share similarities. In model parallelism, linear layers are divided into many small matrices, which are then distributed to different GPUs. For pipeline parallelism, different layers of the model are assigned to different GPUs. Specifically, if we have an n-layer transformer, we can assign the $layer_{i}$ of the transformer to the $GPU_{i}$, and so on. During the forward propagation of the model, we need to perform the computation of the $layer_{i}$ on the $GPU_{i}$, then pass the result to the $GPU_{i+1}$. The $GPU_{i+1}$ receives the output from the $GPU_{i}$, performs the computation for that layer and passes the result to the next GPU. This method partitions the parameters, gradients, optimizer, and intermediate results for each layer.
流水线并行：流水线并行[98]和模型并行有相似之处。在模型并行中，线性层被划分为许多小矩阵，然后分布到不同的 GPU 上。对于流水线并行，模型的各个层被分配到不同的 GPU 上。具体来说，如果我们有一个 n 层 Transformer，我们可以将 Transformer 的 $layer_{i}$ 分配给 $GPU_{i}$ ，依此类推。在模型的正向传播过程中，我们需要在 $GPU_{i}$ 上执行 $layer_{i}$ 的计算，然后将结果传递给 $GPU_{i+1}$ 。 $GPU_{i+1}$ 接收来自 $GPU_{i}$ 的输出，执行该层的计算，并将结果传递给下一个 GPU。这种方法为每一层的参数、梯度、优化器和中间结果进行分区。

In recent years, to pre-train extremely large language models, some research [99] has begun to utilize 16-bit floating-point numbers (FP16) to reduce memory usage and communication overhead. FP16 has a smaller numerical range and lower precision in effective digits [100; 38], but computations tend to be faster than FP32. In general model training, FP32 is often used as the default representation for training parameters. However, in actual model training, the number of parameters in a model typically does not exceed the order of thousands, well within the numerical range of FP16. To improve computational speed, we can convert from FP32 to FP16. During parameter updates, the amount of the parameter is roughly equal to the gradient multiplied by the learning rate. The minimum value of FP16 is on the order of 1e-5. As the product of the gradient and learning rate is already well below the representation range of FP16, the parameter update would result in loss, known as underflow. Therefore, we represent the parameter update obtained by multiplying the gradient by the learning rate as FP32. We cannot directly add this high-precision parameter update to a lower-precision model, as this would still result in floating-point underflow. Consequently, we need to save an additional single-precision parameter on the optimizer. To accelerate both forward and backward passes in the model, half-precision parameters and gradients are used and passed to the optimizer for updating. The optimizer’s update quantity is saved as FP32, and we accumulate it effectively through a temporarily created FP32 parameter in the optimizer. After effective accumulation, it is then converted back to FP16 parameters.
近年来，为了预训练极其庞大的语言模型，一些研究[99]已经开始使用 16 位浮点数（FP16）来减少内存使用和通信开销。FP16 的数值范围较小，有效数字的精度较低[100; 38]，但计算速度通常比 FP32 快。在一般模型训练中，FP32 通常用作训练参数的默认表示。然而，在实际模型训练中，模型的参数数量通常不超过数千的量级，完全在 FP16 的数值范围内。为了提高计算速度，我们可以将 FP32 转换为 FP16。在参数更新期间，参数的量大致等于梯度乘以学习率。FP16 的最小值约为 1e-5。由于梯度和学习率的乘积已经远低于 FP16 的表示范围，参数更新会导致损失，称为下溢。因此，我们将通过将梯度乘以学习率获得的参数更新表示为 FP32。 无法直接将此高精度参数更新添加到低精度模型中，因为这仍然会导致浮点下溢。因此，我们需要在优化器上保存一个额外的单精度参数。为了加速模型中的前向和反向传播，使用半精度参数和梯度，并将它们传递给优化器进行更新。优化器的更新量以 FP32 保存，并通过优化器中临时创建的 FP32 参数有效地累加。在有效累加后，将其转换回 FP16 参数。

The parameters in the optimizer are at least twice as many as the model parameters, and a study [101]proposes the idea of moving the optimizer’s parameters from the GPU to the CPU. Although GPU computation is much faster than CPU, the question arises whether offloading this operation could become a bottleneck for the overall training speed of the model optimizer. In reality, we utilize ZeRO3. After the optimization with ZeRO3, the size of the parameters, gradients, and optimizer is reduced to 1/n of the number of GPUs. By binding one GPU to multiple CPUs, we effectively lower the computational load on each CPU.
优化器中的参数至少是模型参数的两倍，一项研究[101]提出了将优化器参数从 GPU 迁移到 CPU 的想法。尽管 GPU 计算速度远快于 CPU，但这个问题出现了，即这种操作是否可能成为模型优化器整体训练速度的瓶颈。实际上，我们使用了 ZeRO3。经过 ZeRO3 优化后，参数、梯度和优化器的大小减少到 GPU 数量的 1/n。通过将一个 GPU 绑定到多个 CPU 上，我们有效地降低了每个 CPU 的计算负载。

Memory operations are typically asynchronous. Thus, We can send a request to the memory in advance and then proceed with other computations. After completing other computations, we come back to handle the memory request. This two-step operation is used in the forward propagation process of model training. We need to obtain the parameters of $layer_{i}$ through a gather operation. After obtaining the parameters of $layer_{i}$, in the forward propagation process of $layer_{i}$, we proactively retrieve the parameters of $layer_{i+1}$ through an asynchronous fetch. Once the forward propagation calculation for $layer_{i}$ is completed, the parameters for $layer_{i+1}$ have been obtained and are stored in the GPU. We can then immediately proceed with the forward propagation calculation, and so on.
内存操作通常是异步的。因此，我们可以提前向内存发送请求，然后继续进行其他计算。完成其他计算后，我们返回处理内存请求。这种两步操作用于模型训练的前向传播过程。我们需要通过收集操作获取 $layer_{i}$ 的参数。在获取 $layer_{i}$ 的参数后，在 $layer_{i}$ 的前向传播过程中，我们通过异步获取主动检索 $layer_{i+1}$ 的参数。一旦完成 $layer_{i}$ 的前向传播计算， $layer_{i+1}$ 的参数已经获取并存储在 GPU 中。然后我们可以立即进行前向传播计算，等等。

In order to support the backward propagation of the model, All intermediate results in the GPU memory need to be saved during the forward propagation of the model. To optimize this process, a checkpoint mechanism, which does not save all intermediate results in the GPU memory but only retains certain checkpoint points is utilized.
为了支持模型的反向传播，在模型的前向传播过程中，GPU 内存中的所有中间结果都需要被保存。为了优化这一过程，采用了一种检查点机制，该机制不保存 GPU 内存中的所有中间结果，而只保留某些检查点。

The diagram below illustrates a simplified structure of a transformer. Each transformer block takes a model input, undergoes complex computations through attention and feed-forward processes, and produces the overall output of that layer. We keep only the input of each major layer in the transformer as our checkpoint.
以下图示展示了变压器的简化结构。每个变压器块接收模型输入，通过注意力和前馈过程进行复杂计算，并产生该层的整体输出。我们只保留变压器中每个主要层的输入作为我们的检查点。

During the backward propagation process, how do we compute the gradients of the linear layers within each major layer? We can perform a technique called recomputation, which involves re-executing the forward pass of each major layer during the backward propagation process. We temporarily obtain the inputs of the linear layers within each major layer, and the intermediate results obtained can be used for backward propagation. Once the backward propagation for that layer is complete, we can discard the checkpoint and the temporarily recomputed intermediate results of the linear layers within the model from the GPU memory.
在反向传播过程中，我们如何计算每个主要层中线性层的梯度？我们可以执行一种称为重新计算的技术，这涉及到在反向传播过程中重新执行每个主要层的正向传播。我们暂时获得每个主要层中线性层的输入，获得的中间结果可用于反向传播。一旦该层的反向传播完成，我们可以从 GPU 内存中丢弃检查点和模型中线性层的暂时重新计算的中间结果。

Assuming we have a transformer with 24 layers, each layer containing four to five linear layers, using the checkpoint mechanism reduces the originally required storage of 120 intermediate results to only 24 intermediate results.
假设我们有一个包含 24 层的 transformer，每层包含四到五个线性层，使用检查点机制将原本所需的 120 个中间结果减少到仅 24 个中间结果。

The core concept of supervised fine-tuning involves adjusting the model in a supervised manner on the basis of large-scale pre-training, enhancing its capability to better adapt to the specific requirements of the target task. In the process of SFT, it is necessary to prepare a labeled dataset for the target task, which includes input text along with corresponding labels. Instruction tuning is a commonly used technique in the fine-tuning process of LLMs and can be considered as a specific form of SFT. It involves further training LLMs on a dataset composed of (instruction, output) pairs, focusing on enhancing the capabilities and controllability of large language models by understanding and following human instructions. We compiled commonly used instruction tuning datasets, as illustrated in Table 3.
监督微调的核心概念涉及在大规模预训练的基础上，以监督方式调整模型，增强其更好地适应目标任务特定要求的能力。在 SFT 过程中，有必要为目标任务准备一个标记的数据集，其中包括输入文本及其相应的标签。指令调整是在LLMs微调过程中常用的一种技术，可以被视为 SFT 的一种特定形式。它涉及在由（指令，输出）对组成的集合上进一步训练LLMs，通过理解和遵循人类指令，专注于增强大型语言模型的能力和可控性。我们编制了常用的指令调整数据集，如表 3 所示。

Due to LLMs being pre-trained on massive and diverse internet data, even though the training data undergoes some preprocessing, it is still challenging to guarantee the absence of biased or harmful content in terabyte-scale training datasets. Despite LLMs demonstrating impressive performance across various natural language processing tasks, they frequently exhibit behaviors diverging from human intent. This includes generating false information, producing expressions with bias or misleading content, and so on [93; 109]. To address these issues of LLMs displaying behaviors beyond human intent, alignment tuning becomes crucial [93; 110].
由于LLMs在大量且多样化的互联网数据上进行了预训练，尽管训练数据经过了一些预处理，但在 PB 级规模的训练数据集中仍然难以保证不存在有偏见或有害的内容。尽管LLMs在各种自然语言处理任务中表现出令人印象深刻的性能，但它们经常表现出与人类意图不一致的行为。这包括生成虚假信息，产生带有偏见或误导性内容的表达等[ 93; 109]。为了解决LLMs显示超出人类意图的行为问题，对齐调整变得至关重要[ 93; 110]。

In general, alignment tuning aims to meet the following three criteria: being helpful, honest, and harmless.
一般来说，对齐调整旨在满足以下三个标准：有帮助、诚实、无害。

Helpful: The concept of helpfulness revolves around whether the model-generated output proves genuinely beneficial for a specific task or inquiry. In the realm of natural language processing, the model’s generated text or responses should furnish valuable information, positively impacting the user’s requirements or task objectives.
有用：有用性的概念围绕模型生成的输出是否真正对特定任务或查询有益。在自然语言处理领域，模型生成的文本或响应应提供有价值的信息，积极影响用户的需求或任务目标。

Honest: Honesty entails whether the model-generated output is authentic and reliable. The model should produce information consistent with facts, steering clear of fabrication or distortion. This contributes to maintaining user trust in the authenticity of the model’s outputs.
诚实：诚实意味着模型生成的输出是否真实可靠。模型应产生与事实一致的信息，避免虚构或扭曲。这有助于维持用户对模型输出真实性的信任。

Harmless: Harmlessness is concerned with whether the model-generated output poses no harm to users or society. The model should refrain from generating content that is harmful, offensive, or perilous, ensuring its utilization remains safe for all relevant stakeholders.
无害：无害性关注模型生成的输出是否对用户或社会造成伤害。模型应避免生成有害、冒犯性或危险的内容，确保其使用对所有相关利益相关者都是安全的。

In training LLMs, a noteworthy approach to alignment tuning is based on Reinforcement Learning with Human Feedback (RLHF) [93]. This method involves collecting human feedback data to train a reward model (RM) for reinforcement learning. The RM serves as the reward function during reinforcement learning training, and algorithms such as Proximal Policy Optimization (PPO) [111] are employed to fine-tune the LLM. In this context, LLM is considered as the policy, and the action space is considered as the vocabulary of the LLM.
在训练LLMs时，一种值得注意的对齐调整方法是基于人类反馈的强化学习（RLHF）[93]。该方法涉及收集人类反馈数据以训练奖励模型（RM）用于强化学习。RM 在强化学习训练期间作为奖励函数，并采用如近端策略优化（PPO）[111]等算法来微调LLM。在这种情况下，LLM被视为策略，动作空间被视为LLM的词汇表。

Currently, large-scale PLMs such as ChatGPT [93; 19] continue to grow in scale. However, for the majority of researchers, conducting full fine-tuning on consumer-grade hardware has become cost-prohibitive and impractical. Unlike SFT and alignment tuning, the objective of parameter-efficient tuning is to reduce computational and memory overhead. This method involves fine-tuning only a small or additional subset of model parameters while keeping the majority of pre-trained parameters fixed, thereby significantly lowering computational and storage costs. It is noteworthy that state-of-the-art parameter-efficient tuning techniques have achieved performance levels comparable to full fine-tuning. Some common parameter-efficient tuning methods include Low-Rank Adaptation (LoRA) [112], Prefix Tuning [113] and P-Tuning [114; 115]. The adoption of these methods enables efficient model tuning even in resource-constrained environments, offering feasibility and efficiency for practical applications.
目前，大型 PLM 如 ChatGPT[93; 19]的规模仍在不断扩大。然而，对于大多数研究人员来说，在消费级硬件上进行全面微调已成为成本高昂且不切实际的做法。与 SFT 和 alignment tuning 不同，参数高效调优的目标是降低计算和内存开销。这种方法仅对模型参数的小部分或额外子集进行微调，同时保持大部分预训练参数不变，从而显著降低计算和存储成本。值得注意的是，最先进的参数高效调优技术已经达到了与全面微调相当的性能水平。一些常见的参数高效调优方法包括低秩适应（LoRA）[112]、前缀调优[113]和 P-Tuning[114; 115]。采用这些方法即使在资源受限的环境中也能实现高效的模型调优，为实际应用提供了可行性和效率。

With the rise of LLMs, parameter-efficient tuning has garnered increasing attention, with LoRA being widely employed in the latest releases of LLMs. LoRA [112] and its related advancements [116; 117] are noteworthy and deserve attention.
随着LLMs的兴起，参数高效调优越来越受到关注，LoRA 在LLMs的最新版本中得到广泛应用。LoRA[112]及其相关进展[116; 117]值得关注。

To enhance the safety and responsibility of LLMs, the integration of additional safety techniques during fine-tuning is essential. This encompasses three primary techniques, applicable to both SFT and RLHF phases.
为了提高LLMs的安全性和责任感，在微调过程中集成额外的安全技术至关重要。这包括三个主要技术，适用于 SFT 和 RLHF 阶段。

Supervised Safety Fine-Tuning: In this technique, labelers are tasked with generating demonstration data that incorporates high safety risk adversarial prompts. This handcraft safety demonstration data is then incorporated into the SFT phase, thereby augmenting the model’s capacity to manage safety risks.
监督安全微调：在这种技术中，标注员负责生成包含高安全风险对抗性提示的演示数据。然后，这种手工安全演示数据被纳入 SFT 阶段，从而增强模型管理安全风险的能力。

Safety RLHF: This technique employs the same or even more aggressive adversarial prompts to query the models. The safest response exhibiting refusal behavior is then used to train a safety reward model within the RLHF framework.
安全 RLHF：该技术采用相同甚至更激进的对抗性提示来查询模型。然后使用表现出拒绝行为的最高安全响应在 RLHF 框架内训练一个安全奖励模型。

Safety Context Distillation: This technique employs context distillation [118] by initially prefixing safety preprompts, like “You are a safe and responsible assistant,” to adversarial prompts. This process yields safer generated responses. The model is then fine-tuned on these safer demonstration data but without the inclusion of the safety pre-prompts. This safety distillation further enhances the model’s safety capabilities.
安全上下文蒸馏：该技术通过在对抗性提示前添加安全预提示，如“你是一个安全且负责任的助手”，来采用上下文蒸馏[118]。这个过程产生了更安全的生成响应。然后，模型在这些更安全的演示数据上微调，但不包括安全预提示。这种安全蒸馏进一步增强了模型的安全性。

The evaluation of large models’ capabilities requires appropriate datasets for validation. Here, we introduce several commonly used datasets for testing purposes. Considering multimodal large models, typical datasets for computer vision include ImageNet [121] and Open Images [122]. In addition to the commonly used GLUE [123] and SuperGLUE [124] for LLMs, MMLU [125] is highly competitive in testing comprehensive capability. If your model primarily uses Chinese language, then CMMLU [126], as a benchmark for Chinese large models, should also be considered, and XTREME [127] and XTREME-R [128] are suitable choices for multilingual large models. For assessing mathematical knowledge capabilities, there are datasets such as MATH [129] and GSM8K [130], while HumanEval [131] and MBPP [132] can serve as benchmarks for code generation. For common sense reasoning tests in daily human life and work, the following datasets are available: HelloSwag [133], PIQA [134], BoolQ [135], SIQA [136], WinoGrande [137], ARC [138], and OpenBookQA [139]. For medical knowledge, there are datasets such as MedQA-USMLE [140] and MedMCQA [141].
大型模型能力评估需要适当的验证数据集。在此，我们介绍了几个常用的测试数据集。考虑到多模态大型模型，计算机视觉的典型数据集包括 ImageNet [121] 和 Open Images [122]。除了常用的 GLUE [123] 和 SuperGLUE [124] 外，MMLU [125] 在测试综合能力方面具有高度竞争力。如果您的模型主要使用中文，那么作为中文大型模型的基准，CMMLU [126] 也应考虑，XTREME [127] 和 XTREME-R [128] 是多语言大型模型的合适选择。在评估数学知识能力方面，有如 MATH [129] 和 GSM8K [130] 的数据集，而 HumanEval [131] 和 MBPP [132] 可作为代码生成的基准。对于日常人类生活和工作中常识推理测试，以下数据集可用：HelloSwag [133]、PIQA [134]、BoolQ [135]、SIQA [136]、WinoGrande [137]、ARC [138] 和 OpenBookQA [139]。在医学知识方面，有如 MedQA-USMLE [140] 和 MedMCQA [141] 的数据集。

Currently, LLMs interact with humans in the form of questions and answers. Compared to the fragmented and ambiguous information returned by traditional searches, LLMs provide more realistic and efficient question-and-answer results that align with human habits. Therefore, the evaluation of ODQA (Open Domain Question Answering) [142] capability is essential. The performance of open-domain question answering greatly affects user experience. Commonly used datasets for testing include SquAD [143] and Natural Questions [144], with F1 score and Exact-Match accuracy (EM) as evaluation metrics. However, note that the method of word matching may have certain issues, such as when a factually correct answer is not in the golden answer list. Therefore, human evaluation seems to be necessary, and literature [145] has conducted detailed research on this matter.
目前，LLMs 以问答的形式与人类互动。与传统搜索返回的碎片化和模糊信息相比，LLMs 提供更符合人类习惯的更真实、更高效的问答结果。因此，评估 ODQA（开放域问答）[ 142] 能力至关重要。开放域问答的性能极大地影响用户体验。常用的测试数据集包括 SquAD [ 143] 和 Natural Questions [ 144]，F1 分数和精确匹配准确率（EM）作为评估指标。然而，请注意，单词匹配的方法可能存在某些问题，例如当事实正确的答案不在黄金答案列表中时。因此，似乎需要进行人工评估，文献[ 145] 对此进行了详细研究。

As an emerging and hot research field, LLMs must pay attention to their potential security threats, prevent malicious use or vulnerabilities to malicious attacks, and address any potential long-term issues that may pose a threat to human development. Additionally, red teaming in various domains is necessary to critically assess and test the model, identifying vulnerabilities, biases, inaccuracies, and areas for safety improvement.
作为一个新兴和热门的研究领域，LLMs必须关注其潜在的安全威胁，防止恶意使用或漏洞被恶意攻击利用，并解决可能对人类发展构成威胁的任何潜在长期问题。此外，在各个领域进行红队行动是必要的，以批判性地评估和测试模型，识别漏洞、偏差、不准确之处以及安全改进的领域。

Potential bias:The training data for LLMs may contain potential biases, such as gender or race. Security assessments need to address whether the model generates or amplifies these biases and how to reduce or correct them. Reference [146] discusses in detail the causes of bias in LLMs and the serious consequences that may arise. Reference [147] extensively studies how pre-trained language models generate harmful content to what extent, and how to use controlled text generation algorithms to prevent the generation of toxic content. CHBias [148] is a Chinese dataset that can be used to evaluate and mitigate the bias problem of LLMs.
潜在偏差：LLMs的训练数据可能存在潜在偏差，例如性别或种族。安全评估需要解决模型是否产生或放大这些偏差，以及如何减少或纠正它们。参考文献[146]详细讨论了LLMs偏差的原因以及可能出现的严重后果。参考文献[147]广泛研究了预训练语言模型在多大程度上生成有害内容，以及如何使用受控文本生成算法来防止生成有毒内容。CHBias[148]是一个可用于评估和减轻LLMs偏差问题的中文数据集。

Privacy protection: LLMs may come into contact with a large amount of user data, such as text and images, during the training process. Security assessments need to ensure the effective protection of user data privacy to prevent leaks and misuse. Reference [149] conducted research on models like ChatGPT and found that it is possible to extract training data effectively from these models. Reference [150] provides a solution by proposing a framework called DEPN (Detect and Editing Privacy Neurons) to detect and edit privacy neurons in pre-trained language models. It also introduces a privacy neuron aggregator to eliminate privacy information in a batch-processing manner, effectively reducing the leakage of privacy data while maintaining model performance.
隐私保护：LLMs 在训练过程中可能会接触到大量用户数据，如文本和图像。安全评估需要确保用户数据隐私的有效保护，以防止泄露和滥用。参考文献[149]对 ChatGPT 等模型进行了研究，发现可以从这些模型中有效地提取训练数据。参考文献[150]提出了一种名为 DEPN（检测和编辑隐私神经元）的框架作为解决方案，用于检测和编辑预训练语言模型中的隐私神经元。它还引入了一个隐私神经元聚合器，以批量处理的方式消除隐私信息，在保持模型性能的同时，有效减少隐私数据的泄露。

Adversarial attacks: LLMs may be susceptible to adversarial attacks, such as input tampering, intentional misinformation, or generating false information. Security assessments need to consider the robustness of the model, i.e., its ability to withstand such attacks. As mentioned in reference [151], LLMs still have "jailbreak" risks, where users can manipulate the model to generate toxic content using specific input methods like role-playing or adding special suffixes as studied in the referenced paper. Especially when using open-source pre-trained models, any vulnerabilities in the pre-training models regarding adversarial attacks are inherited as well. Reference [152] provides a solution to mitigate the harm caused by these vulnerabilities.
对抗攻击：LLMs可能容易受到对抗攻击，如输入篡改、故意传播错误信息或生成虚假信息。安全评估需要考虑模型的鲁棒性，即其抵御此类攻击的能力。如参考文献[151]中所述，LLMs仍然存在“越狱”风险，用户可以通过角色扮演或添加特殊后缀等特定输入方法操纵模型生成有害内容，正如参考文献中研究的那样。特别是在使用开源预训练模型时，预训练模型中关于对抗攻击的任何漏洞也会被继承。参考文献[152]提供了一种减轻这些漏洞造成的损害的解决方案。

Automated evaluation and manual evaluation play crucial roles in Language Model (LLM) research. Automated evaluation typically involves using various metrics and indicators to quantify the performance of models, such as BIEU [153], ROUGE [154], and BERTSScore [155], which can measure the accuracy of LLM-generated content. These metrics can help researchers quickly assess model performance on large-scale data and compare different models. However, automated evaluation also has limitations as it cannot fully capture the complexity of language understanding and generation. Research in reference [156] has shown that manual evaluation is more reliable for some open-ended generation tasks. Manual evaluation typically involves human annotators subjectively judging and assessing the quality of model-generated outputs. This evaluation method can help reveal how models perform in specific tasks or scenarios and identify subtle issues and errors that automated evaluation may overlook. However, manual evaluation also faces challenges such as high time costs and subjectivity. Therefore, it is often necessary to combine the strengths of automated and manual evaluation to comprehensively assess the performance of language models.
自动评估和人工评估在语言模型（LLM）研究中起着关键作用。自动评估通常涉及使用各种指标和指标来量化模型的性能，例如 BIEU [153]、ROUGE [154] 和 BERTSScore [155]，这些指标可以衡量LLM生成内容的准确性。这些指标可以帮助研究人员快速评估大规模数据上的模型性能并比较不同模型。然而，自动评估也存在局限性，因为它无法完全捕捉语言理解和生成的复杂性。参考文献[156]的研究表明，对于某些开放式生成任务，人工评估更为可靠。人工评估通常涉及人类标注员主观判断和评估模型生成输出的质量。这种方法可以帮助揭示模型在特定任务或场景中的表现，并识别自动评估可能忽略的细微问题和错误。然而，人工评估也面临诸如时间成本高和主观性等挑战。 因此，通常需要结合自动评估和人工评估的优势，全面评估语言模型的表现。

Knowledge Distillation [175] refers to transferring knowledge from a cumbersome (teacher) model to a smaller (student) model that is more suitable for deployment. This is achieved by fitting the soft targets of the two models, as soft targets provide more information than gold labels. Initially, the calculation for model distillation involved only fitting the outputs from the last layer of both the teacher and student models [176]. PKD [177] improves this process by computing the mean-square loss between normalized hidden states, allowing the student model to learn from multiple intermediate layers of the teacher model. In order to discover more intermediate representations suitable for knowledge distillation, Jiao et al. [178] proposed Tiny BERT. This enables the student model to learn from the embedding layer and attention matrices of the teacher model.
知识蒸馏[175]是指将知识从笨重的（教师）模型转移到更小、更适合部署的（学生）模型。这是通过拟合两个模型的软目标实现的，因为软目标比金标签提供更多信息。最初，模型蒸馏的计算仅涉及拟合教师和学生模型最后层的输出[176]。PKD[177]通过计算归一化隐藏状态之间的均方损失来改进这一过程，使学生模型能够从教师模型的多个中间层中学习。为了发现更多适合知识蒸馏的中间表示，Jiao 等人[178]提出了 Tiny BERT。这使得学生模型能够从教师模型的嵌入层和注意力矩阵中学习。

Model pruning involves removing redundant portions from the parameter matrices of large models. It is divided into unstructured pruning and structured pruning. Unstructured pruning involves removing individual connections or weights in a neural network without adhering to any specific structural pattern. In structured pruning, specific structural patterns or units within a neural network are pruned or removed. Gordon et al. [179] compared the effects of unstructured and structured pruning on the BERT model. They found that the effectiveness of unstructured pruning significantly decreases as the pruning ratio increases, while in structured pruning, 30-40% of the weights can be discarded without affecting BERT’s universality. Different structures in the model can be structurally pruned. Michel et al. [180] pruned attention heads and found that ablating one head often positively impacts the performance of WMT and BERT. They proposed a gradient-based metric for evaluating the importance of attention heads to enhance pruning effectiveness. Fan et al. [179] performed layer pruning by extending dropout from weights to layers. During training, they randomly dropped layers and achieved good inference results by selecting sub-networks with any desired depth during testing.
模型剪枝涉及从大型模型的参数矩阵中移除冗余部分。它分为无结构剪枝和结构剪枝。无结构剪枝涉及在不遵循任何特定结构模式的情况下从神经网络中移除单个连接或权重。在结构剪枝中，神经网络中的特定结构模式或单元被剪枝或移除。Gordon 等人[179]比较了无结构和结构剪枝对 BERT 模型的影响。他们发现，随着剪枝比的增加，无结构剪枝的有效性显著降低，而在结构剪枝中，可以丢弃 30-40%的权重而不影响 BERT 的通用性。模型中的不同结构可以进行结构剪枝。Michel 等人[180]剪除了注意力头，发现移除一个头通常对 WMT 和 BERT 的性能有积极影响。他们提出了一种基于梯度的指标来评估注意力头的重要性，以增强剪枝的有效性。Fan 等人[179]通过将 dropout 从权重扩展到层来执行层剪枝。 在训练过程中，他们随机丢弃层，并在测试时通过选择任何所需深度的子网络实现了良好的推理结果。

The fundamental idea behind model quantization is to reduce the number of floating-point bits used in numerical calculations within a large model network, thereby decreasing storage and computation costs. This involves converting floating-point operations into fixed-precision operations. However, as precision decreases, the model’s loss gradually increases, and when precision drops to 1 bit, the model’s performance experiences a sudden decline. To address the optimization challenges introduced by low-precision quantization, Bai et al. [181] proposed BinaryBERT. They initially trained a half-sized ternary model and then initialized a binary model with the ternary model through weight splitting. Finally, they fine-tuned the binary model. This approach yielded better results for the binary model compared to training a binary model from scratch.
模型量化背后的基本思想是在大型模型网络中减少数值计算中使用的浮点位数量，从而降低存储和计算成本。这涉及到将浮点运算转换为定点运算。然而，随着精度的降低，模型的损失逐渐增加，当精度降低到 1 位时，模型的性能会突然下降。为了解决低精度量化引入的优化挑战，Bai 等人[181]提出了 BinaryBERT。他们最初训练了一个半大小的三元模型，然后通过权重分割用三元模型初始化二进制模型。最后，他们对二进制模型进行了微调。与从头开始训练二进制模型相比，这种方法为二进制模型带来了更好的结果。

The basic idea of weight sharing is to use the same set of parameters for multiple parts of a LLM. Instead of learning different parameters for each instance or component, the model shares a common set of parameters across various parts. Weight sharing helps reduce the number of parameters that need to be learned, making the model more computationally efficient and reducing the risk of overfitting, especially in situations where there is limited data. ALBERT [182] uses the Cross-layer parameter-sharing strategy to effectively reduce the number of parameters of the model, and can achieve better training results than the baseline with the same parameter number.
权重共享的基本思想是为LLM的多个部分使用相同的参数集。而不是为每个实例或组件学习不同的参数，模型在各个部分之间共享一组公共参数。权重共享有助于减少需要学习的参数数量，使模型在计算上更高效，并降低过拟合的风险，尤其是在数据有限的情况下。ALBERT [182] 使用跨层参数共享策略有效地减少了模型的参数数量，并且在使用相同参数数量的基线模型中可以达到更好的训练结果。

Low-rank decomposition methods are crucial in the field of model compression, as they allow for the creation of more compact models with fewer parameters. This reduction in model size is particularly beneficial for deploying neural networks on resource-constrained devices, improving efficiency during inference. Chen et al. [183] performed a low-rank decomposition on the input matrix, enabling matrix operations within the large model to occur at a lower-rank level, effectively reducing the computational workload. From the results, their proposed method, DRONE, not only ensures the inference performance of the large model but also achieves an acceleration ratio of more than 1.3 times compared to the baseline method. The specific choice of low-rank decomposition method depends on the architecture of the neural network and the requirements of the target application.
低秩分解方法在模型压缩领域至关重要，因为它们允许创建具有更少参数的更紧凑模型。这种模型尺寸的减少对于在资源受限的设备上部署神经网络特别有益，提高了推理效率。Chen 等人[183]对输入矩阵进行了低秩分解，使得在大模型中进行矩阵操作可以在较低秩级别进行，从而有效地减少了计算工作量。从结果来看，他们提出的 DRONE 方法不仅确保了大型模型的推理性能，而且与基线方法相比，实现了超过 1.3 倍的加速比。低秩分解方法的具体选择取决于神经网络的架构和目标应用的要求。