Entropy Law: The Story Behind Data Compression and LLM Performance
熵定律：数据压缩和LLM性能背后的故事

The relationship between language modeling and data compression has long intrigued researchers (Shannon, 1948, 1951). Pandey (2024) has identified a data-dependant scaling law that takes data’s gzip compressibility into consideration. Besides, recent empirical studies have confirmed that language models can act as versatile data compressors (Delétang et al., 2023), and the intelligence of LLMs can be quantified by their capacity for text compression (Huang et al., 2024). Let a text corpus be generated from an underlying distribution $\rho$. A lossless compression algorithm $\mathcal{C}$ is then expected to encode a text sequence $x_{1:n}$ into a bitstream $\mathcal{C}(x_{1:n})$ of minimal length, ensuring that $x_{1:n}$ can be perfectly recovered from $\mathcal{C}(x_{1:n})$. The expected number of bits of an optimal $\mathcal{C}$ is equal to $\mathbb{E}_{x\sim\rho}[-\log_{2}\rho(x)]=\mathbb{E}_{x\sim\rho}[-\sum_{i=1}^{n}\log_{2}\rho(x_{i}|x_{1:i-1})]$ (Shannon, 1948). The underlying distribution $\rho$ is usually unknown in reality, but it can be estimated by a language model $\rho_{\text{model}}$. Then the expected number of bits of an optimal $\mathcal{C}$ can be updated:
语言建模与数据压缩之间的关系长期以来一直吸引着研究人员（Shannon, 1948, 1951）。Pandey（2024）已经确定了一种依赖于数据的缩放定律，该定律考虑了数据的 gzip 可压缩性。此外，最近的实证研究证实，语言模型可以作为多功能的数据压缩器（Delétang et al, 2023），而LLMs的智能可以通过其文本压缩能力来量化（Huang et al, 2024）。假设一个文本语料库是从一个基础分布 $\rho$ 生成的。然后，一个无损压缩算法 $\mathcal{C}$ 应该将文本序列 $x_{1:n}$ 编码成最小长度的比特流 $\mathcal{C}(x_{1:n})$ ，确保可以从 $\mathcal{C}(x_{1:n})$ 完美恢复 $x_{1:n}$ 。最优 $\mathcal{C}$ 的预期比特数等于 $\mathbb{E}_{x\sim\rho}[-\log_{2}\rho(x)]=\mathbb{E}_{x\sim\rho}[-\sum_{i=1}^{n}\log_{2}\rho(x_{i}|x_{1:i-1})]$ （Shannon, 1948）。基础分布 $\rho$ 在现实中通常是未知的，但可以通过语言模型 $\rho_{\text{model}}$ 来估计。然后，最优 $\mathcal{C}$ 的预期比特数可以更新为：

Equation 1 is the cross-entropy loss employed in training LLMs, thereby establishing a coherent relationship between LLMs and information compression. This foundational insight paves the way for this work.
方程 1 是训练LLMs时使用的交叉熵损失，从而在LLMs和信息压缩之间建立了一种连贯的关系。这一基础性见解为本工作的开展铺平了道路。

Large Language Models (LLMs) have recently gained significant attention from academia and industry. LLM alignment, which includes supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), has emerged as a crucial technique for adapting LLMs to end tasks using natural language instructions (Zhao et al., 2023; Wang et al., 2023c). Alignment is performed using instruction datasets consisting of multiple (Instruction, Output) pairs, which require LLMs to follow the instructions and generate corresponding outputs. Early explorations have focused on constructing or expanding instruction datasets through methods such as crowd-sourcing (Wang et al., 2022; Köpf et al., 2024), self-instruction (Taori et al., 2023; Peng et al., 2023; Wang et al., 2023b), or the combination of existing datasets (Wang et al., 2023a; Ivison et al., 2023). Fine-tuned LLMs on these datasets have demonstrated promising capabilities to adhere to instructions across various contexts and align with human expectations.
大型语言模型（LLMs）最近在学术界和工业界引起了广泛关注。LLM对齐，包括监督微调（SFT）和基于人类反馈的强化学习（RLHF），已成为使用自然语言指令将LLMs适应终端任务的关键技术（Zhao 等，2023；Wang 等，2023c）。对齐是使用包含多个（指令，输出）对的指令数据集进行的，这些数据集要求LLMs遵循指令并生成相应的输出。早期的探索集中在通过众包（Wang 等，2022；Köpf 等，2024）、自我指令（Taori 等，2023；Peng 等，2023；Wang 等，2023b）或现有数据集的组合（Wang 等，2023a；Ivison 等，2023）等方法构建或扩展指令数据集。在这些数据集上微调的LLMs展示了在各种情境下遵循指令并符合人类期望的有希望的能力。

A growing body of research has emphasized the importance of selecting appropriate data for LLM alignment, which can prevent potential quality issues and optimize computational resource allocation. As a prominent example, Lima (Zhou et al., 2023) has demonstrated superior performance by carefully crafting only 1,000 high-quality samples for SFT, highlighting the crucial importance of data quality. The current literature on selecting alignment data has focused on selecting samples according to individual sample quality, which can be categorized into heuristic methods (Shen, 2024) and model-based methods (Chen et al., 2023; Lu et al., 2023; Liu et al., 2023; Li et al., 2023, 2024; Du et al., 2023). Heuristic methods typically employ specific criteria, such as response length (Shen, 2024), to guide data selection. On the other hand, model-based methods adopt various strategies to leverage the capabilities of established language models for evaluating sample quality. For example, IFD (Li et al., 2023) measures the change in response loss when instructions are removed, and selects those with the most significant changes. Building upon IFD, SuperFiltering (Li et al., 2024) introduces a lightweight proxy model for a more efficient calculation of the IFD score. In addition, other model-based methods employ proprietary LLMs to assess data quality. In a pioneering work, AlpaGasus (Chen et al., 2023) uses ChatGPT directly to assign data quality scores to samples, while #InsTag (Lu et al., 2023) proposes assigning tags to each sample using ChatGPT and evaluates sample quality based on the number of tags. DEITA (Liu et al., 2023) uses ChatGPT-generated data to train two Llama-based scorers, assigning complexity and quality scores to each sample, and ultimately selecting samples with the highest hybrid scores. However, existing methods are mainly designed to pick data based on sample-wise quality measurements, which are usually weak in reflecting the overall dataset quality. In this paper, we focus on the relation between performance and dataset quality, which can be efficiently measured by data compression metrics.
越来越多的研究强调选择适当的数据进行LLM对齐的重要性，这可以防止潜在的质量问题并优化计算资源分配。作为一个突出的例子，Lima（Zhou 等，2023）通过精心制作仅 1,000 个高质量样本用于 SFT，展示了卓越的性能，突显了数据质量的关键重要性。目前关于选择对齐数据的文献主要集中在根据单个样本质量选择样本，这可以分为启发式方法（Shen，2024）和基于模型的方法（Chen 等，2023；Lu 等，2023；Liu 等，2023；Li 等，2023，2024；Du 等，2023）。启发式方法通常采用特定标准，如响应长度（Shen，2024），来指导数据选择。另一方面，基于模型的方法采用各种策略来利用已建立的语言模型的能力来评估样本质量。例如，IFD（Li 等，2023）通过测量删除指令时响应损失的变化，并选择那些变化最显著的样本。 在 IFD 的基础上，SuperFiltering（Li 等，2024）引入了一种轻量级代理模型，以更高效地计算 IFD 分数。此外，其他基于模型的方法采用专有的LLMs来评估数据质量。在一项开创性工作中，AlpaGasus（Chen 等，2023）直接使用 ChatGPT 为样本分配数据质量分数，而#InsTag（Lu 等，2023）提出使用 ChatGPT 为每个样本分配标签，并根据标签数量评估样本质量。DEITA（Liu 等，2023）使用 ChatGPT 生成的数据训练了两个基于 Llama 的评分器，为每个样本分配复杂度和质量分数，最终选择混合分数最高的样本。然而，现有方法主要设计用于基于样本质量测量来挑选数据，这通常难以反映整体数据集的质量。在本文中，我们关注性能与数据集质量之间的关系，这可以通过数据压缩指标高效地测量。

In general, we maintain an information redundancy state $\pi_{\mathcal{D}}$ that evaluates the “information gain” of each sample. Intuitively, data with high intra-sample information redundancy are unlikely to have good global diversity. For example, a sample with repeated patterns or echoed conversation turns usually has low education value in LLM training. Thus, we initialize this state by calculating the sample-level compression ratio for the entire dataset $\mathcal{D}$. In each iteration, We select $K_{1}$ samples with the lowest scores in $\pi_{\mathcal{D}}$ to form an initial candidate pool $\mathcal{D}_{K_{1}}$, which provides a good set for subsequent local selection.
一般来说，我们保持一个信息冗余状态 $\pi_{\mathcal{D}}$ ，用于评估每个样本的“信息增益”。直观上，具有高样本内信息冗余的数据不太可能具有良好的全局多样性。例如，具有重复模式或重复对话轮次的样本通常在LLM训练中具有较低的教育价值。因此，我们通过计算整个数据集的样本级压缩比来初始化此状态 $\mathcal{D}$ 。在每次迭代中，我们选择 $K_{1}$ 个在 $\pi_{\mathcal{D}}$ 中得分最低的样本来形成一个初始候选池 $\mathcal{D}_{K_{1}}$ ，这为后续的局部选择提供了一个良好的集合。

Since the global selection does not well consider the mutual relations among samples, we further conduct local selection to pick diverse samples. To ensure good computational efficiency, we introduce a coarse-grained selection phase to narrow the candidate pool into a smaller one with $K_{2}$ samples. We first compute the compression ratio of a merged set that adds each sample in $\mathcal{D}_{K_{1}}$ to the selected set $\mathcal{D}^{\prime}$. We use this score to update the information redundancy state $\pi_{\mathcal{D}}$ to better indicate the current information gain of these samples. Based on the scores of the samples in $\mathcal{D}_{K_{1}}$, we select $K_{2}$ samples with the lowest scores. These samples form a small subset for final fine-grained selection, where each sample has good diversity with the selected dataset $\mathcal{D}^{\prime}$.
由于全局选择未能很好地考虑样本之间的相互关系，我们进一步进行局部选择以挑选多样化的样本。为了确保良好的计算效率，我们引入了一个粗粒度选择阶段，将候选池缩小到一个包含 $K_{2}$ 个样本的较小池。我们首先计算将 $\mathcal{D}_{K_{1}}$ 中的每个样本添加到已选择集 $\mathcal{D}^{\prime}$ 中的合并集的压缩比。我们使用此分数更新信息冗余状态 $\pi_{\mathcal{D}}$ ，以更好地指示这些样本的当前信息增益。基于 $\mathcal{D}_{K_{1}}$ 中样本的分数，我们选择分数最低的 $K_{2}$ 个样本。这些样本形成一个用于最终细粒度选择的小子集，其中每个样本与已选择数据集 $\mathcal{D}^{\prime}$ 具有良好的多样性。

Although the above stage ensures that the candidate pool has distinct information from the selected set, the information redundancy among the samples within this pool is not measured. Thus, we aim to pick further samples from this subset that are diverse from each other. Concretely, we initialize a local selected set $\mathcal{D}_{K_{3}}$, and compute the compression ratio of the union of $\mathcal{D}_{K_{3}}$ and each sample in $\mathcal{D}_{K_{2}}$. We add the sample with the lowest compression ratio into $\mathcal{D}_{K_{3}}$, and remove it from $\mathcal{D}_{K_{2}}$. By repeating this process, we obtain a small subset that contains samples not only different from the selected set $\mathcal{D}^{\prime}$ but also distinct from each other. We conduct the three stages above until the size of $\mathcal{D}^{\prime}$ reaches a predefined data budget $m$. In our method, the entire selection process is quite easy to implement since it is model-free, and can be accelerated using multiple threads. It can select data efficiently and effectively from a large candidate pool for high-quality LLM training.
尽管上述阶段确保了候选池中的信息与已选集合中的信息不同，但未测量该池内样本之间的信息冗余。因此，我们旨在从该子集中挑选彼此不同的进一步样本。具体来说，我们初始化一个本地选定集合 $\mathcal{D}_{K_{3}}$ ，并计算 $\mathcal{D}_{K_{3}}$ 和 $\mathcal{D}_{K_{2}}$ 中每个样本的并集的压缩比。我们将压缩比最低的样本添加到 $\mathcal{D}_{K_{3}}$ 中，并将其从 $\mathcal{D}_{K_{2}}$ 中移除。通过重复此过程，我们获得了一个小子集，其中包含不仅与选定集合 $\mathcal{D}^{\prime}$ 不同的样本，而且彼此不同的样本。我们进行上述三个阶段，直到 $\mathcal{D}^{\prime}$ 的大小达到预定义的数据预算 $m$ 。在我们的方法中，整个选择过程非常容易实现，因为它是无模型的，并且可以使用多线程加速。它可以从大型候选池中高效且有效地选择高质量LLM训练的数据。

In this section, we aim to demonstrate the proposed entropy law. Specifically, we have plotted the model performance of Mistral-7B and Llama-3-8B concerning data compression ratio and training loss in Figure 3 and 4, respectively. Besides, we plot entropy-law curves by fitting the results. From the two figures, we can draw the following analysis:
在本节中，我们旨在展示所提出的熵定律。具体来说，我们在图 3 和图 4 中分别绘制了 Mistral-7B 和 Llama-3-8B 的模型性能与数据压缩比和训练损失的关系。此外，我们通过拟合结果绘制了熵定律曲线。从这两幅图中，我们可以得出以下分析：

Relationship between model performance, data compression ratio, and training loss In Figure 3(a) and Figure 4(a), LLMs trained on data with a lower compression ratio typically exhibit enhanced performance. Since the learning process of LLMs is highly relevant to information compression, we can regard LLMs as data compressors. Then the data with a lower compression ratio means a higher knowledge amount, which is more beneficial to the compressors. Besides, a lower compression ratio usually corresponds a higher training loss, as illustrated in Figures 3(b) and 4(b). This is because data resistant to compression carries more knowledge, posing a greater challenge for LLMs to absorb the encapsulated knowledge.
模型性能、数据压缩比和训练损失之间的关系 在图 3(a)和图 4(a)中，LLMs在较低压缩比的数据上训练通常表现出更好的性能。由于LLMs的学习过程与信息压缩高度相关，我们可以将LLMs视为数据压缩器。那么较低压缩比的数据意味着更高的知识量，这对压缩器更有利。此外，较低的压缩比通常对应较高的训练损失，如图 3(b)和图 4(b)所示。这是因为抗压缩的数据携带更多的知识，对LLMs吸收封装的知识构成了更大的挑战。

Model performance interpretation with entropy law Considering the three methods with comparable compression ratios and training loss, namely Random, Cluster, and Perplexity, the corresponding model performances are close. This phenomenon may seem counter-intuitive, given the distinct criteria used for data selection. However, it aligns with the predictions of our proposed entropy law: when the average data quality, training loss, and data compression ratio are similar, the model performance is expected to be comparable as well. Thus, the entropy law has the potential to serve as a criterion for predicting the model’s performance on data, thereby guiding the training of LLMs.
利用熵定律解释模型性能 考虑到三种具有可比压缩比和训练损失的方法，即随机、聚类和困惑度，对应的模型性能是接近的。鉴于用于数据选择的不同标准，这一现象可能看起来违反直觉。然而，这与我们提出的熵定律的预测是一致的：当平均数据质量、训练损失和数据压缩比相似时，模型性能也预计是可比的。因此，熵定律有可能作为预测模型在数据上性能的标准，从而指导LLMs的训练。

Practical application of entropy law Incremental version update of training data is a common setting in practical LLM development. Usually, the training data amount remains relatively stable, with only a minor portion undergoing modification. We have conducted incremental training data update experiments in real scenarios, with results depicted in Figure 5. Due to confidentiality, only the relative order of the results is provided. Guided by entropy law, assuming the data quality $Q$ does not significantly decay after each incremental update, we can expect a model performance improvement with a decreased data compression ratio. This expectation is supported by the results from data versions $x_{1}$ to $x_{4}$ in Figure 5 since their compression ratios are decreased after each incremental update. However, the data version $x_{5}$ exhibits an abnormal increase in the loss and data compression ratio, which serves as an early indicator of potential model performance degradation due to a decline in training data consistency. This prediction is further confirmed by subsequent post-training model performance evaluations, as illustrated in Figure 5. Thus, the entropy Law can be utilized as a guideline for LLM training to identify potential risks of experimental failure without training the model on the full dataset until convergence. This is particularly significant given the substantial costs associated with training an LLM.
熵定律的实际应用 训练数据的增量版本更新是实际开发中的常见设置。通常，训练数据量保持相对稳定，只有一小部分进行修改。我们在实际场景中进行了增量训练数据更新实验，结果如图 5 所示。由于保密原因，仅提供结果的相对顺序。在熵定律的指导下，假设数据质量在每次增量更新后不会显著下降，我们可以预期模型性能会随着数据压缩比的降低而提高。图 5 中数据版本 $x_{1}$ 到 $x_{4}$ 的结果支持了这一预期，因为它们的压缩比在每次增量更新后都降低了。然而，数据版本 $x_{5}$ 表现出损失和数据压缩比的异常增加，这作为训练数据一致性下降导致模型性能潜在退化的早期指标。后续的训练后模型性能评估进一步证实了这一预测，如图 5 所示。 因此，熵定律可以作为LLM训练的指导方针，用于在不将模型训练到整个数据集收敛的情况下识别实验失败的潜在风险。鉴于训练LLM所需的巨大成本，这一点尤为重要。