Local Distortion Aware Efficient Transformer Adaptation for Image Quality Assessment
局部失真感知高效 Transformer 适配用于图像质量评估

By the success of deep learning in many computer vision tasks, different approaches utilize deep learning for IQA: early CNN-based [11, 1, 2] and recently transformer-based methods [24, 13]. Modern CNN-based models commonly posit that initial stages within the network encapsulate low-level spatial characteristics, whereas subsequent stages are indicative of higher-level semantic features [27]. Based on this, Su et al. [11] put forth a method wherein multi-scale features and semantic features are extracted from images using the ResNet architecture [15]. Subsequently, they endeavor to capture local distortion information from the multi-scale features and generate weights utilizing semantic features to serve as a quality prediction target network. Lastly, the target network assimilates the aggregated local distortion features as input to predict image quality.
随着深度学习在许多计算机视觉任务中的成功，不同的方法利用深度学习进行图像质量评估：早期的基于 CNN 的方法[11, 1, 2]以及最近的基于 Transformer 的方法[24, 13]。现代的基于 CNN 的模型通常认为网络中的初始阶段包含低级别的空间特征，而后续阶段则显示出高级别的语义特征[27]。基于此，Su 等人[11]提出了一种方法，其中使用 ResNet 架构[15]从图像中提取多尺度特征和语义特征。随后，他们努力从多尺度特征中捕捉局部失真信息，并利用语义特征生成权重以作为质量预测目标网络。最后，目标网络将聚合的局部失真特征作为输入来预测图像质量。

Although CNNs capture the local structure of the image, they are well known for missing to capture non-local information and having strong locality bias. On the contrary, Vision Transformer (ViT) [28] has strong capability in modeling the non-local dependencies among features of the image, thus transformer-based methods demonstrate great potential in dealing with the image quality assessment. Golestaneh, Dadsetan, and Kitani [24] proposed a method that utilizes CNNs to extract the perceptual features as inputs to the Transformer encoder. Ke et al. [12] and Qin et al. [13] directly send image patches as inputs to the Transformer encoder.
虽然 CNN 能够捕捉图像的局部结构，但它们因未能捕捉非局部信息和具有较强的局部性偏差而广为人知。相反，Vision Transformer（ViT）在建模图像特征的非局部依赖性方面具有较强的能力，因此基于 Transformer 的方法在处理图像质量评估方面表现出巨大潜力。Golestaneh、Dadsetan 和 Kitani 提出了一种方法，利用 CNN 提取感知特征作为 Transformer 编码器的输入。Ke 等人和 Qin 等人则直接将图像块作为输入发送到 Transformer 编码器。

Recently, the parameter capacities of vision models are undergoing a rapid expansion, scaling from 480M parameters of EfficientNet-based [16] to 22B parameters of Transformer-based counterparts [18]. As a consequence, their requisition for training data and training techniques is similarly expanding. In terms of this, these models are commonly trained on large-scale labeled datasests [8, 29] in a supervised or self-supervised manner. Moreover, some works [10] adopt large-scale multi-modal data (e.g., image-text pairs) for training, which leads to even more powerful visual representations. In this work, we could take advantage of these well pretrained image models and adapt them efficiently to solve IQA tasks.
最近，视觉模型的参数容量正在迅速扩展，从基于 EfficientNet 的 480M 参数扩展到基于 Transformer 的 22B 参数。因此，它们对训练数据和训练技术的需求也在不断增长。从这方面来看，这些模型通常在大型标注数据集上进行有监督或自监督的训练。此外，一些工作[10]采用大型多模态数据（例如，图像-文本对）进行训练，从而产生更强大的视觉表征。在这项工作中，我们可以利用这些经过良好预训练的图像模型，并有效地适应它们来解决图像质量评估（IQA）任务。

In the field of NLP, efficient model adaptation techniques involve adding or modifying a limited number of parameters of the model, as limiting the dimension of the optimization problem can prevent catastrophic forgetting. Conventional arts [11] typically adopt full-tuning in the downstream tasks. Rare attention has been drawn to the field of efficient adaptation, especially in the field of vision Transformers. However, with the surge of large-scale pretrained models, the conventional paradigm is inevitably limited by the huge computational burden, thus some works [20, 21] migrate the efficient model adaptation techniques that appeared in NLP to CV.
在自然语言处理领域，高效的模型适应技术涉及增加或修改模型的少量参数，因为限制优化问题的维度可以防止灾难性遗忘。传统技术[11]通常在下游任务中采用全调优。对于高效适应领域，尤其是在视觉 Transformer 领域，鲜有关注。然而，随着大规模预训练模型的兴起，传统范式不可避免地受到巨大计算负担的限制，因此一些工作[20, 21]将自然语言处理中出现的高效模型适应技术迁移到计算机视觉领域。

Due to the paucity of labeled data for training, IQA methods are unable to realize their full potential. Previous works [11, 24] commonly full-finetune the whole network trained on ImageNet-1K, but the model and data are insufficiently large. In this work, we propose employing efficient model adaptation techniques to adapt large-scale pretrained models to IQA tasks.
由于缺乏用于训练的标注数据，图像质量评估（IQA）方法无法充分发挥其潜力。先前的研究[11, 24]通常在 ImageNet-1K 上训练整个网络并进行完整微调，但模型和数据规模都不够大。在这项工作中，我们提出采用高效的模型适应技术，以将大规模预训练模型适配于 IQA 任务。

In an effort to further improve the efficiency of pretrained model adaptation and customize it for IQA tasks, we devise a transformer-based adaptation efficient framework, namely LOcal Distortion Aware efficient transformer adaptation (LoDa). As depicted in Figure 2, the framework of LoDa is composed of two components. The initial component incorporates a large-scale pretrained Vision Transformer (ViT) [28]. The second component comprises a pretrained CNN tasked with extracting multi-scale features from the input image. Moreover, it integrates a local distortion extractor responsible for capturing local distortion features from the extracted multi-scale features. Subsequently, a local distortion-aware injector is employed to procure corresponding local distortion tokens, which are similar to tokens of the ViT model, and then infuse them for the later stage.
为了进一步提高预训练模型适应的效率，并使其定制化用于图像质量评估任务，我们设计了一种基于 Transformer 的高效适应框架，称为局部失真感知高效 Transformer 适应（LoDa）。如图 2 所示，LoDa 的框架由两个组件组成。初始组件包含一个大规模预训练的 Vision Transformer（ViT）。第二个组件包括一个预训练的 CNN，负责从输入图像中提取多尺度特征。此外，它整合了一个局部失真提取器，负责从提取的多尺度特征中捕捉局部失真特征。随后，使用一个局部失真感知注入器来获取相应的局部失真标记，这些标记类似于 ViT 模型的标记，然后将它们注入到后续阶段。

Specifically, upon receiving an input image, our process initiates by directing it to a pretrained CNN to extract multi-scale features. Subsequently, these mult-scale feature maps are individually routed into separate local distortion extractors, generating distinct local distortion features. These local distortion features are then reshaped and concatenated to create multi-scale distortion tokens for later interaction. Simultaneously, the input image is further input into the patch embedding layer of the pretrained ViT. Here, the image undergoes division into non-overlapping $16\times 16$ patches, which are then flattened and transformed into $D$-dimensional tokens by the patch embedding layer. Subsequently, these tokens are added with position embeddings. After this process, the tokens, acting as queries, are coupled with the multi-scale distortion tokens and are subjected to cross-attention. This results in the extraction of similar local distortion features from the multi-scale distortion tokens, which are subsequently injected into the tokens of the ViT, thereby enhancing the distortion-related information encompassed by these tokens. Following this, the tokens, along with the augmented distortion features, traverse through $L$ transformer encoder layers and cross-attention blocks. Finally, the CLS token acquired from the ViT serves as the input to the quality regressor, enabling the derivation of the final quality score.
具体而言，在接收到输入图像后，我们的流程首先通过预训练的 CNN 提取多尺度特征。随后，这些多尺度特征图被分别引导至各自的局部畸变提取器，生成不同的局部畸变特征。这些局部畸变特征随后被重新整形并连接起来，以创建用于后续交互的多尺度畸变标记。同时，输入图像还被输入到预训练 ViT 的 patch 嵌入层。在这里，图像被划分为不重叠的 $16\times 16$ 块，然后这些块被扁平化并通过 patch 嵌入层转换为 $D$ 维标记。接着，这些标记被加上位置嵌入。经过此过程后，作为查询的标记与多尺度畸变标记结合，并接受交叉注意力措施。这导致从多尺度畸变标记中提取出相似的局部畸变特征，这些特征随后被注入到 ViT 的标记中，从而增强了这些标记所包含的失真相关信息。 随后，这些标记连同增强的失真特征一起穿越 $L$ 个 Transformer 编码器层和交叉注意力块。最后，从 ViT 获取的 CLS 标记作为质量回归器的输入，从而能够得出最终的质量评分。

It is noteworthy that during adaptation, only the local distortion extractor modules, local distortion aware injectors and the head are trainable, but the weights of the pretrained ViT encoder and pretrained CNN are frozen.
值得注意的是，在适配过程中，只有局部失真提取模块、局部失知注入器和头部是可训练的，而预训练的 ViT 编码器和预训练的 CNN 的权重是被冻结的。

The majority of large-scale pretrained models [8, 7, 17, 10] are grounded in the Transformer architecture [23], renowned for its robust capacity to model non-local dependencies among perceptual features within an image. However, these models exhibit a weak inductive bias. Conversely, CNNs excel at capturing the local structure of an image, exhibiting a strong locality bias, but they falter in capturing non-local information [24, 13]. Nonetheless, IQA is highly reliant on both local and non-local features [24, 13]. In the absence of abundant labeled data [3, 14], the adaptation of large-scale pretrained models to IQA may suffer from a deficiency in local structure and inductive bias. This deficiency can, however, be mitigated by leveraging the capabilities of CNNs [30]. In light of these considerations, we propose the exploitation of the local structure and inductive bias derived from pretrained CNNs to strengthen the adaptation of large-scale pretrained models for IQA without altering their original architecture.
大多数大规模预训练模型[8, 7, 17, 10]基于 Transformer 架构[23]，以其强大的能力著称，能够建模图像感知特征中的非局部依赖关系。然而，这些模型表现出了较弱的归纳偏置。相反，CNN 在捕捉图像的局部结构方面表现优异，展现出了强烈的局部性偏置，但在捕捉非局部信息时表现不佳[24, 13]。然而，IQA 高度依赖于局部和非局部特征[24, 13]。在缺乏丰富标注数据的情况下[3, 14]，将大规模预训练模型适应于 IQA 可能会因局部结构和归纳偏置的不足而受影响。然而，这种不足可以通过充分利用 CNN 的能力来缓解[30]。鉴于这些考虑，我们提议利用预训练 CNN 的局部结构和归纳偏置来加强大规模预训练模型在 IQA 中的适应性，而不改变其原有架构。

As shown in Figure 2, with the given input image $I\in\mathbb{R}^{H\times W\times C}$, the pretrained CNN will output a set of multi-scale features $F^{j}\in\mathbb{R}^{b\times c_{j}\times m_{j}\times n_{j}}$, where $j$ denotes the $j$-th block of CNN, $b$ denotes the batch size and $c_{j}$, $m_{j}$ and $n_{j}$ denote the channel size, width, and height of the $j$-th features, respectively. The reason why we extract multi-scale features is that semantic features extracted from the last layer merely represent holistic image content [11]. In order to capture local distortions in the real world, we propose to extract multi-scale features $F^{j}$ through a local distortion extractor. As illustrated in Figure 3(a), we use sequential trainable $1\times 1$ and $3\times 3$ convolutional layers to project them into equal dimensions and further extract image quality-related local distortion features and inductive bias from the multi-scale features. Since the initial features of pretrained CNNs have relatively large dimensions, to keep adaptation efficiency, we use average pooling to pool the extracted features into a smaller size. Let $\bar{F}^{j}\in\mathbb{R}^{b,c,m,n}$ denote the output feature after sending $F^{j}$ to the convolutions and pooling. Next, we flatten and concatenate $\bar{F}^{j}$ and obtain the multi-scale distortion tokens $F_{msd}\in\mathbb{R}^{b,\sum_{j}(m\times n),c}$, as the input for the local distortion injector.
如图 2 所示，给定输入图像 $I\in\mathbb{R}^{H\times W\times C}$ ，预训练的 CNN 将输出一组多尺度特征 $F^{j}\in\mathbb{R}^{b\times c_{j}\times m_{j}\times n_{j}}$ ，其中 $j$ 表示 CNN 的第 $j$ 个块， $b$ 表示批量大小，而 $c_{j}$ 、 $m_{j}$ 和 $n_{j}$ 分别表示第 $j$ 个特征的通道大小、宽度和高度。我们提取多尺度特征的原因在于，最后一层提取的语义特征仅代表整体图像内容[11]。为了捕捉真实世界中的局部失真，我们提出通过一个局部失真提取器来提取多尺度特征 $F^{j}$ 。如图 3(a)所示，我们使用序列可训练的 $1\times 1$ 和 $3\times 3$ 卷积层将它们投影到相同维度，并进一步从多尺度特征中提取与图像质量相关的局部失真特征和归纳偏差。由于预训练 CNN 的初始特征维度相对较大，为了保持适应效率，我们使用平均池化将提取的特征池化为较小的尺寸。让 $\bar{F}^{j}\in\mathbb{R}^{b,c,m,n}$ 表示将 $F^{j}$ 送入卷积和池化后的输出特征。 接下来，我们将 $\bar{F}^{j}$ 展平并进行连接，获得多尺度畸变令牌 $F_{msd}\in\mathbb{R}^{b,\sum_{j}(m\times n),c}$ ，作为局部畸变注入器的输入。

A direct approach to infusing multi-scale distortion tokens into tokens of large-scale pretrained ViT models involves a simple addition of the features with the tokens. Nevertheless, it should be noted that an image token in ViT corresponds to a $16\times 16$ patch of the original image, which might not align with the scale of the multi-scale distortion features. To address this misalignment, we introduce to use cross-attention mechanism, which enables to query features akin to the image token of ViT from the multi-scale distortion features. Subsequently, the queried features are adeptly combined with the image tokens, ensuring a coherent and effective integration of the distortion information.
一种将多尺度失真标记直接注入到大规模预训练 ViT 模型的标记中的方法是对特征与标记进行简单的相加。然而，需要注意的是，ViT 中的图像标记对应于原始图像的 $16\times 16$ 补丁，这可能与多尺度失真特征的尺度不一致。为了解决这一不匹配，我们引入了使用交叉注意力机制的方法，这使得能够从多尺度失真特征中查询类似于 ViT 图像标记的特征。随后，被查询的特征与图像标记巧妙地结合，确保失真信息的连贯和有效集成。

As illustrated in Figure 3(b), after passing the input image $I$ to large-scale pretrained ViT, assuming that $F_{vit}^{i}\in\mathbb{R}^{b,l,d}$ denote the token of $i^{th}$ block of the ViT (including CLS token and image token). We take $F_{vit}^{i}$ as query $Q_{i}$ and multi-scale distortion tokens $F_{msd}$ as key $K_{i}$ and value $V_{i}$ of multi-head cross-attention (MHCA) to obtain multi-scale distortion tokens that are similar to $F_{vit}^{i}$ from $F_{msd}$:
如图 3(b) 所示，将输入图像 $I$ 输入到大规模预训练的 ViT 中，假设 $F_{vit}^{i}\in\mathbb{R}^{b,l,d}$ 表示 ViT 的第 $i^{th}$ 块的标记（包括 CLS 标记和图像标记）。我们将 $F_{vit}^{i}$ 作为查询 $Q_{i}$ ，将多尺度畸变标记 $F_{msd}$ 作为多头交叉关注（MHCA）的键 $K_{i}$ 和值 $V_{i}$ ，从 $F_{msd}$ 中获得与 $F_{vit}^{i}$ 类似的多尺度畸变标记：

Then, the queried multi-scale distortion tokens are added with ViT tokens $F_{vit}^{i}$, which can be written as Eqn. 2:
随后，查询到的多尺度失真标记与 ViT 标记 $F_{vit}^{i}$ 相加，可以表示为公式 2：

where $s^{i}$ represents a trainable vector designed to strike a balance between the output of the attention layer and the input feature $F_{vit}^{i}$. To facilitate this balance, $s^{i}$ is initialized to a value close to 0. This specific initialization strategy ensures that the feature distribution of $F_{vit}^{i}$ remains unchanged despite the injection of queried multi-scale distortion features, thereby allowing for more effective utilization of the pretrained weights of ViT in the adaptation process.
其中 $s^{i}$ 表示一个可训练的向量，旨在平衡注意力层的输出和输入特征 $F_{vit}^{i}$ 。为了促进这种平衡， $s^{i}$ 被初始化为一个接近于零的值。这一特定的初始化策略确保了 $F_{vit}^{i}$ 的特征分布在注入多尺度失真特征后保持不变，从而允许在适应过程中更有效地利用 ViT 的预训练权重。

Due to the channel dimension of the large-scale pretrained vision transformer being relatively large (768 for ViT-B), directly using this for extra MHCA will bring a tremendous amount of parameters and computational overhead, which is not consistent with efficient model adaptation. Inspired by adapter [26] in NLP, we propose to down project ViT token $F_{vit}^{i}$ and multi-scale distortion features $F_{msd}$ to a smaller dimension $r$,
由于大规模预训练视觉 Transformer 的通道维度相对较大（例如 ViT-B 为 768），直接使用这种方式进行额外的 MHCA 会导致参数量和计算开销大幅增加，这与高效模型适应不符。受 NLP 中适配器[26]的启发，我们提出将 ViT 标记 $F_{vit}^{i}$ 和多尺度失真特征 $F_{msd}$ 降维到较小的维度 $r$ 。

where $f$ is a trainable MLP layer, performs the projection of ViT token $F_{vit}^{i}$ and multi-scale distortion features $F_{msd}$ into $\tilde{F}_{vit}^{i}\in\mathbb{R}^{b,l,r}$ and $\tilde{F}_{msd}\in\mathbb{R}^{b,\sum_{j}(m\times n),r}$, respectively. Notably, it is $\tilde{F}_{vit}^{i}$ and $\tilde{F}_{msd}$ that take on the roles of query $Q_{i}$, key $K_{i}$ and value $V_{i}$ within MHCA, instead of $F_{vit}^{i}$ and $F_{msd}$. Lastly, we up-project the result from cross-attention by a trainable MLP layer into the dimension of ViT tokens.
其中 $f$ 是一个可训练的 MLP 层，执行 ViT 令牌 $F_{vit}^{i}$ 和多尺度失真特征 $F_{msd}$ 到 $\tilde{F}_{vit}^{i}\in\mathbb{R}^{b,l,r}$ 和 $\tilde{F}_{msd}\in\mathbb{R}^{b,\sum_{j}(m\times n),r}$ 的投影。值得注意的是，在 MHCA 中， $\tilde{F}_{vit}^{i}$ 和 $\tilde{F}_{msd}$ 承担了查询 $Q_{i}$ 、键 $K_{i}$ 和值 $V_{i}$ 的角色，而不是 $F_{vit}^{i}$ 和 $F_{msd}$ 。最后，我们通过一个可训练的 MLP 层将交叉注意力的结果向上投影到 ViT 令牌的维度。

With the output CLS token of ViT, we feed it into a single-layer regressor head to obtain the quality score. A PLCC-induced loss is employed for training. Assuming there are $m$ images on the training batch and the predicted quality scores $\tilde{y}=\{\tilde{y}_{1},\tilde{y}_{2},\dots,\tilde{y}_{m}\}$ and corresponding label $y=\{y_{1},y_{2},\dots,y_{m}\}$, the PLCC-induced loss is defined as:
利用 ViT 的输出 CLS token，我们将其输入到单层回归头以获得质量评分。采用 PLCC 诱导的损失进行训练。假设训练批次中有 $m$ 张图像，预测的质量评分为 $\tilde{y}=\{\tilde{y}_{1},\tilde{y}_{2},\dots,\tilde{y}_{m}\}$ ，对应的标签为 $y=\{y_{1},y_{2},\dots,y_{m}\}$ ，PLCC 诱导损失定义为：

where $\tilde{a}$ and $a$ are the mean values of $\tilde{y}$ and $y$, respectively.
其中 $\tilde{a}$ 和 $a$ 分别是 $\tilde{y}$ 和 $y$ 的平均值。

The performance comparison over the State-of-the-art (SOTA) BIQA methods is shown in Table I. Our model outperforms the existing SOTA methods [37, 38, 39, 2, 40, 41, 1, 11, 12, 24, 13, 42, 3, 14] by a significant margin on these datasets of both synthetically and authentically distorted images. Since images on various datasets span a wide variety of image contents and distortion types, it is still challenging to consistently achieve the leading performance on all of them.
与现有的最先进（SOTA）BIQA 方法的性能比较如表 I 所示。我们的模型在合成和真实失真的图像数据集上均显著优于现有的 SOTA 方法 [37, 38, 39, 2, 40, 41, 1, 11, 12, 24, 13, 42, 3, 14]。由于各个数据集中的图像涵盖了广泛的图像内容和失真类型，因此在所有数据集上持续保持领先性能仍然具有挑战性。

Specifically, ours surpass traditional methods (e.g., ILNIQE [37] and BRISQUE [38]) and earlier learning-based methods (e.g., TIQA [40] and HyperIQA [11] by a large margin. For LIQE [42] that utilized a large-scale pretrained vision-language model, multi-task labels, and full fine-tuning on multiple datasets simultaneously, LoDa still outperforms on both large synthetical and authentical datasets, i.e., KADID10k and KonIQ10k. Compared with current SOTA methods that required extra pertaining (e.g., DEIQT [13], Re-IQA [3] and QPT-ResNet50 [14]), LoDa obtains competitive or higher results, showing the powerful effectiveness of adaptation of large-scale pretrained models. Correspondingly, the top performance on the largest synthetical datasets KADID-10k confirms the superiority of fusing the multi-scale distortion features from CNN into ViT model.
具体来说，我们的方法大幅超越了传统方法（如 ILNIQE [37]和 BRISQUE [38]）以及早期的学习型方法（如 TIQA [40]和 HyperIQA [11]）。对于使用了大规模预训练视觉-语言模型、多任务标签以及同时对多个数据集进行完整微调的 LIQE [42]，LoDa 在大型合成和真实数据集上仍然表现优异，即 KADID10k 和 KonIQ10k。与需要额外预训练（如 DEIQT [13]、Re-IQA [3]和 QPT-ResNet50 [14]）的当前 SOTA 方法相比，LoDa 获得了具有竞争力或更高的结果，显示出大规模预训练模型适应性的强大效果。相应地，在最大的合成数据集 KADID-10k 上表现出的顶级性能证实了将多尺度畸变特征从 CNN 融合到 ViT 模型中的优势。

We further compare the generalizability of LoDa against competitive BIQA models in a cross-dataset setting following [13]. Training is performed on one specific dataset, and testing is performed on a different dataset without any finetuning or parameter adaptation. The experimental results in terms of the medians of SRCC on four datasets are reported in Table II. As observed, LoDa achieves the best performance on all datasets. These results manifest the strong generalization capability of LoDa.
我们进一步比较了 LoDa 在跨数据集设置中的泛化能力，针对具有竞争力的 BIQA 模型，遵循[13]进行比较。训练在一个特定数据集上进行，测试则在不同的数据集上进行，且没有任何微调或参数调整。实验结果以四个数据集上的 SRCC 中值形式报告在表 II 中。可以观察到，LoDa 在所有数据集上都获得了最佳性能。这些结果表明了 LoDa 强大的泛化能力。

One of the key properties of our proposed methods is the efficient adaptation of large-scale pretrained models, which allows our model to achieve a competing performance to SOTA BIQA methods. To demonstrate the effectiveness of using large-scale pretrained models in our proposed models, we employ different pretrained weights, including ImageNet-1K pretrained weights [7], ImageNet-21K pretrained weights [7], MAE pretrained weights [43], and Multi-Modal pretrained weights [44], and evaluate them on relatively large synthetical and authentical datasets, KADID-10k and KonIQ-10k. The experimental results are detailed in Table III. The transition from weights pretrained on ImageNet-1K to those pretrained on ImageNet-21K yields more benefits for our model, as the scale of pretraining data expansively increases. Besides, While MAE also employs ImageNet-1K pretraining, it distinguishes itself from supervised ImageNet-1K pretraining by embracing a more potent self-supervised pretraining approach, which also confers substantial advantages upon our model. However, our model faces challenges in effectively leveraging multi-modal pretrained weights. One plausible explanation is that multi-modal pretrained models may prioritize the abstract concepts inherent within images, a focus that diverges from the demands of IQA tasks. Since multi-modal pretrained weights contain more information than single-modal pretrained ones, how to apply these models to the IQA tasks will also be an important topic and we will commit to conducting further research on this.
我们提出的方法的关键特性之一是能够高效地适应大规模预训练模型，这使得我们的模型能够在 BIQA 方法中取得与现有最佳方法相媲美的性能。为了展示在我们提出的模型中使用大规模预训练模型的有效性，我们采用了不同的预训练权重，包括 ImageNet-1K 预训练权重、ImageNet-21K 预训练权重、MAE 预训练权重和多模态预训练权重，并在相对较大的合成和真实数据集 KADID-10k 和 KonIQ-10k 上进行评估。实验结果详见表 III。从 ImageNet-1K 预训练权重过渡到 ImageNet-21K 预训练权重为我们的模型带来了更多的好处，因为预训练数据的规模大幅增加。此外，虽然 MAE 也采用 ImageNet-1K 预训练，但它通过采用更强大的自监督预训练方法，与传统的有监督的 ImageNet-1K 预训练有别，这也为我们的模型带来了显著的优势。然而，我们的模型在有效利用多模态预训练权重方面面临挑战。 一个合理的解释是，多模态预训练模型可能更注重图像内在的抽象概念，而这种关注与图像质量评估任务的要求有所偏离。由于多模态预训练权重包含的信息要比单模态预训练的多，因此如何将这些模型应用于图像质量评估任务也将是一个重要的课题，我们会致力于对此进行进一步研究。

Moreover, the parameter capacities of large-scale pretrained models are another essential component of our method. To verify the effectiveness of large-scale pretrained model size, we evaluate LoDa with ViT-Tiny/Small/Base, and all ViTs are pretrained with ImageNet-21k. Quantitative results are shown in Table IV. From this, we can observe that with the growth of pretrained backbone sizes, our model can benefit from it and thus achieve better performance. In particular, solely employing ViT-S as the backbone, our method can achieve performance on par with SOTA shown in Table I, which further shows the effectiveness of our method.
此外，大规模预训练模型的参数容量是我们方法的另一个重要组成部分。为了验证大规模预训练模型尺寸的有效性，我们使用 ViT-Tiny/Small/Base 进行 LoDa 评估，所有的 ViT 都是在 ImageNet-21k 上预训练的。定量结果如表 IV 所示。从中可以观察到，随着预训练主干尺寸的增加，我们的模型能够从中受益，进而取得更好的性能。特别是，仅使用 ViT-S 作为主干，我们的方法就能够在表 I 中达到与 SOTA 相当的性能，这进一步显示了我们方法的有效性。

At present, numerous efficient model adaptation methods for large-scale pretrained vision models have emerged, Adapter [26], LoRA [45] and visual prompt tuning (VPT) [20] stands as the exemplars. To demonstrate the effectiveness of our proposed method, we employ linear probing ViT that only fine-tunes the head of ViT, full fine-tuning ViT, Adapter-ViT, LoRA-ViT, and VPT-ViT for IQA task, and compare it with our method on KADID-10K and KonIQ-10k datasets. The experimental results are detailed in Table V. It can be noticed that our model outperforms all of these fine-tuning methods on KADID-10K and KonIQ-10K, especially on KADID-10K, which shows the effectiveness and superiority of fusing CNN multi-scale distortion features into ViT.
目前，已经出现了众多大型预训练视觉模型的高效模型适配方法，Adapter [26]、LoRA [45] 和视觉提示调优（VPT）[20] 就是其中的典范。为了证明我们提出方法的有效性，我们应用线性探测 ViT（仅微调 ViT 的头部）、完全微调的 ViT、Adapter-ViT、LoRA-ViT 以及针对 IQA 任务的 VPT-ViT，并在 KADID-10K 和 KonIQ-10K 数据集上与我们的方法进行比较。实验结果详见表 V。可以注意到，我们的模型在 KADID-10K 和 KonIQ-10K 数据集上均优于所有这些微调方法，尤其是在 KADID-10K 上，这表明将 CNN 多尺度失真特征融合到 ViT 中的有效性和优越性。

This supplementary material presents: (1) additional experimental analysis and quantitative results of the ablation study of LoDa; (2) more visualization of Fourier analysis of vision transformer (ViT) [28] and LoDa.
本补充材料介绍：(1) LoDa 消融研究的额外实验分析和定量结果；(2) 更多视觉转换器（ViT）[28]和 LoDa 傅里叶分析的可视化结果。

In the paper, we show the Fourier analysis of features of ViT and LoDa on the KADID-10k dataset, here we additionally show the Fourier analysis of full-fintuned ViT and LoDa on the KonIQ-10k dataset (average over 128 images) in Figure 5. We can observe the same results on the KonIQ-10k dataset, it further demonstrates that LoDa captures more high-frequency signals and show the effect of our proposed method.