AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection
AnomalyCLIP：面向零样本异常检测的对象无关提示学习

To effectively learn the object-agnostic text prompts, we devise a joint optimization approach that enables the normality and abnormality prompt learning from both global and local perspectives, namely global and local context optimization. The global context optimization aims to enforce that our object-agnostic textual embeddings are matched with the global visual embeddings of images of diverse objects. This helps effectively capture the normal/abnormal semantics from a global feature perspective. The local context optimization is introduced to enable object-agnostic text prompts to concentrate on fine-grained, local abnormal regions from $M$ intermediate layers of the visual encoder, in addition to the global normal/abnormal features. Formally, let $\mathcal{M}$ be the set of intermediate layers used (i.e., $M=|\mathcal{M}|$), our text prompts are learned by minimizing the following glocal loss function:
为了有效学习与对象无关的文本提示，我们设计了一种联合优化方法，该方法能够从全局和局部两个角度进行正常和异常提示的学习，即全局和局部上下文优化。全局上下文优化的目的是确保我们的与对象无关的文本嵌入与不同对象图像的全局视觉嵌入相匹配。这有助于从全局特征的角度有效捕捉正常/异常语义。引入局部上下文优化是为了使与对象无关的文本提示能够集中关注视觉编码器 $M$ 中间层的细粒度局部异常区域，同时兼顾全局正常/异常特征。形式上，设 $\mathcal{M}$ 为所使用的中间层集合（即 $M=|\mathcal{M}|$ ），我们的文本提示通过最小化以下全局-局部损失函数来学习：

where $\lambda$ is a hyperparameter to balance the global and local losses, $L_{global}$ is a cross entropy loss that matches the cosine similarity between the object-agnostic textual embeddings and visual embeddings of normal/abnormal images from auxiliary data, and let $S\in\mathbb{R}^{H\times W}$ be the ground-truth segmentation mask, with $S_{ij}=1$ if the pixel is as an anomaly and $S_{ij}=0$ otherwise, then we have
其中 $\lambda$ 是平衡全局和局部损失的超参数， $L_{global}$ 是交叉熵损失，用于匹配来自辅助数据的正常/异常图像的对象无关文本嵌入和视觉嵌入之间的余弦相似度， $S\in\mathbb{R}^{H\times W}$ 为真实分割掩码，若像素为异常则 $S_{ij}=1$ ，否则 $S_{ij}=0$ ，则有

where $Focal(\cdot,\cdot)$ and $Dice(\cdot,\cdot)$ denote a focal loss (Lin et al., 2017) and a Dice loss (Li et al., 2019) respectively, the operator $[\cdot,\cdot]$ represents the concatenation along with the channel, and $I$ represents the full-one matrix. Since the anomalous regions are typically smaller than the normal ones, we use focal loss to address the imbalance problem. Furthermore, to ensure that the model establishes an accurate decision boundary, we employ the Dice loss to measure the overlaps between the predicted segmentation $S_{n}$/$S_{a}$ and the ground truth mask.
其中 $Focal(\cdot,\cdot)$ 和 $Dice(\cdot,\cdot)$ 分别表示焦点损失（Lin 等人，2017）和 Dice 损失（Li 等人，2019），运算符 $[\cdot,\cdot]$ 表示沿通道的串联， $I$ 表示全一矩阵。由于异常区域通常小于正常区域，我们使用焦点损失来解决不平衡问题。此外，为了确保模型建立准确的决策边界，我们采用 Dice 损失来衡量预测分割 $S_{n}$ / $S_{a}$ 与真实掩码之间的重叠。

To facilitate the learning of a more discriminative textual space via Eq. 2, inspired by Jia et al. (2022) and Khattak et al. (2023), we introduce text prompt tuning to refine the original textual space of CLIP by adding additional learnable token embeddings into its text encoder. Specifically, we first attach randomly initialized learnable token embeddings $t^{\prime}_{m}$ into $T_{m}$, the $m$-th layer of the frozen CLIP text encoder. Then, we concatenate $t^{\prime}_{m}$ and the original token embeddings $t_{m}$ along the dimension of the channel, and forward them to $T_{m}$ to get the corresponding $r^{\prime}_{m+1}$ and $t_{m+1}$. To ensure proper calibration, we discard the obtained $r^{\prime}_{m+1}$ and initialize new learnable token embeddings $t^{\prime}_{m+1}$. Note that even though the output $r^{\prime}_{m+1}$ is discarded, the updated gradients can still be backpropagated to optimize the learnable tokens $t^{\prime}_{m}$ due to the self-attention mechanism. We repeat this operation until we reach the designated layer $M^{\prime}$. During fine-tuning, these learnable token embeddings are optimized to refine the original textual space. More details see Appendix D.
为了通过公式 2 促进更具区分性的文本空间的学习，受 Jia 等人（2022 年）和 Khattak 等人（2023 年）的启发，我们引入了文本提示调优，通过向 CLIP 的文本编码器添加额外的可学习令牌嵌入来优化其原始文本空间。具体而言，我们首先将随机初始化的可学习令牌嵌入 $t^{\prime}_{m}$ 附加到冻结的 CLIP 文本编码器的第 $m$ 层 $T_{m}$ 中。然后，我们沿通道维度连接 $t^{\prime}_{m}$ 和原始令牌嵌入 $t_{m}$ ，并将它们前向传递到 $T_{m}$ 以获得相应的 $r^{\prime}_{m+1}$ 和 $t_{m+1}$ 。为了确保适当的校准，我们丢弃获得的 $r^{\prime}_{m+1}$ 并初始化新的可学习令牌嵌入 $t^{\prime}_{m+1}$ 。请注意，即使输出 $r^{\prime}_{m+1}$ 被丢弃，由于自注意力机制，更新的梯度仍可以反向传播以优化可学习令牌 $t^{\prime}_{m}$ 。我们重复此操作，直到达到指定层 $M^{\prime}$ 。在微调期间，这些可学习令牌嵌入被优化以改进原始文本空间。更多细节见附录 D。

Since the visual encoder of CLIP is originally pre-trained to align global object semantics, the contrastive loss used in CLIP makes the visual encoder produce a representative global embedding for class recognition. Through the self-attention mechanism, the attention map in the visual encoder focuses on the specific tokens highlighted within the red rectangle in Fig 3b. Although these tokens may contribute to global object recognition, they disrupt the local visual semantics, which directly hinders the effective learning of the fine-grained abnormality in our object-agnostic text prompts. We found empirically that a diagonally prominent attention map helps reduce the disturbance from other tokens, leading to improved local visual semantics. Therefore, we propose a mechanism called Diagonally Prominent Attention Map to refine the local visual space, with the visual encoder kept frozen during training. To this end, we replace the original $Q$-$K$ attention in the visual encoder with diagonally prominent attention, such as $Q$-$Q$, $K$-$K$, and $V$-$V$ self-attention schemes. As demonstrated in Fig.3c, Fig.3d, and Fig. 3e, the refined DPAM attention maps are more diagonally prominent, resulting in substantially improved segmentation maps in both original CLIP and our AnomalyCLIP. Compared to CLIP that is based on global features and manually defined text prompts, the text prompts learned by AnomalyCLIP are more fine-grained, enabling substantially more accurate alignment between the normality/abnormality prompt embeddings and the local visual embeddings across four different self-attention schemes. This, in turn, allows AnomalyCLIP to generate accurate $S_{n}$ and $S_{a}$ for the joint optimization in Eq. 2. Unless otherwise specified, AnomalyCLIP utilizes $V$-$V$ self-attention due to its superior overall performance. The performance of different self-attention mechanisms is analyzed in Sec. D. We also provide a detailed explanation about DPAM in Appendix C.
由于 CLIP 的视觉编码器最初是预训练用于对齐全局对象语义的，CLIP 中使用的对比损失使视觉编码器生成用于类别识别的代表性全局嵌入。通过自注意力机制，视觉编码器中的注意力图聚焦于图 3b 中红色矩形内突出显示的特定标记。尽管这些标记可能有助于全局对象识别，但它们扰乱了局部视觉语义，这直接阻碍了我们对象无关文本提示中细粒度异常的有效学习。我们经验性地发现，对角线突出的注意力图有助于减少其他标记的干扰，从而改善局部视觉语义。因此，我们提出了一种称为对角线突出注意力图的机制，以优化局部视觉空间，同时在训练期间保持视觉编码器冻结。为此，我们将视觉编码器中的原始 $Q$ - $K$ 注意力替换为对角线突出的注意力，例如 $Q$ - $Q$ 、 $K$ - $K$ 和 $V$ - $V$ 自注意力方案。如图 3c、图 3d 和图所示。 3e，经过优化的 DPAM 注意力图在斜对角线上更为突出，从而在原始 CLIP 和我们的 AnomalyCLIP 中显著改善了分割图。与基于全局特征和手动定义文本提示的 CLIP 相比，AnomalyCLIP 学习的文本提示更为精细，使得在四种不同的自注意力机制下，正常/异常提示嵌入与局部视觉嵌入之间的对齐更加精确。这进而使 AnomalyCLIP 能够为公式 2 中的联合优化生成准确的 $S_{n}$ 和 $S_{a}$ 。除非另有说明，AnomalyCLIP 采用 $V$ - $V$ 自注意力机制，因其整体性能优越。不同自注意力机制的性能分析见第 D 节。我们还在附录 C 中提供了关于 DPAM 的详细解释。

During training, AnomalyCLIP minimizes the loss in Eq. 2 using an auxiliary AD-related dataset. As for inference, given a test image $x_{i}$, we use the similarity score $P(g_{a},f_{i})$ as the image-level anomaly score, with the anomaly score leaning toward one when the anomaly textual embedding $g_{a}$ is aligned with the global visual embedding $f_{i}$. For pixel-wise predictions, we merge the segmentation $S_{n}$ and $S_{a}$, followed by an interpolation and smoothing operation. Formally, our anomaly score map $M\in\mathbb{R}^{H\times W}$ is computed as $M=G_{\sigma}(Bilinear\_interploation(\frac{1}{2}(I-S_{n}+S_{a})))$, where $G_{\sigma}$ represents a Gaussian filter, and $\sigma$ controls the extent of smoothing.
在训练过程中，AnomalyCLIP 利用一个辅助的 AD 相关数据集最小化公式 2 中的损失。至于推理阶段，给定测试图像 $x_{i}$ ，我们使用相似度分数 $P(g_{a},f_{i})$ 作为图像级别的异常分数，当异常文本嵌入 $g_{a}$ 与全局视觉嵌入 $f_{i}$ 对齐时，异常分数倾向于 1。对于像素级预测，我们融合分割结果 $S_{n}$ 和 $S_{a}$ ，随后进行插值和平滑操作。形式上，我们的异常分数图 $M\in\mathbb{R}^{H\times W}$ 计算为 $M=G_{\sigma}(Bilinear\_interploation(\frac{1}{2}(I-S_{n}+S_{a})))$ ，其中 $G_{\sigma}$ 代表高斯滤波器， $\sigma$ 控制平滑程度。

We conducted extensive experiments on 17 publicly available datasets, covering various industrial inspection scenarios and medical imaging domains (including photography, endoscopy, and radiology) to evaluate the performance of AnomalyCLIP. In industrial inspection, we consider MVTec AD (Bergmann et al., 2019), VisA (Zou et al., 2022), MPDD (Jezek et al., 2021), BTAD (Mishra et al., 2021), SDD (Tabernik et al., 2020), DAGM (Wieler & Hahn, 2007), and DTD-Synthetic (Aota et al., 2023). In medical imaging, we consider skin cancer detection dataset ISIC (Gutman et al., 2016), colon polyp detection datasets CVC-ClinicDB (Bernal et al., 2015), CVC-ColonDB (Tajbakhsh et al., 2015), Kvasir (Jha et al., 2020), and Endo (Hicks et al., 2021), thyroid nodule detection dataset TN3k (Gong et al., 2021), brain tumor detection datasets HeadCT (Salehi et al., 2021), BrainMRI (Salehi et al., 2021), Br35H (Hamada., 2020), and COVID-19 detection dataset COVID-19 (Chowdhury et al., 2020; Rahman et al., 2021). The SOTA competing methods include CLIP (Radford et al., 2021), CLIP-AC (Radford et al., 2021), WinCLIP (Jeong et al., 2023), VAND (Chen et al., 2023), and CoOp (Zhou et al., 2022b). We provide more details about the methods and data pre-processing in Appendix A. The anomaly detection performance is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC). Additionally, average precision (AP) for anomaly detection and AUPRO (Bergmann et al., 2020) for anomaly segmentation are also used to provide more in-depth analysis of the performance.
我们在 17 个公开可用的数据集上进行了广泛的实验，涵盖了各种工业检测场景和医学影像领域（包括摄影、内窥镜和放射学），以评估 AnomalyCLIP 的性能。在工业检测中，我们考虑了 MVTec AD（Bergmann 等，2019）、VisA（Zou 等，2022）、MPDD（Jezek 等，2021）、BTAD（Mishra 等，2021）、SDD（Tabernik 等，2020）、DAGM（Wieler & Hahn，2007）和 DTD-Synthetic（Aota 等，2023）。在医学影像中，我们考虑了皮肤癌检测数据集 ISIC（Gutman 等，2016）、结肠息肉检测数据集 CVC-ClinicDB（Bernal 等，2015）、CVC-ColonDB（Tajbakhsh 等，2015）、Kvasir（Jha 等，2020）和 Endo（Hicks 等，2021）、甲状腺结节检测数据集 TN3k（Gong 等，2021）、脑肿瘤检测数据集 HeadCT（Salehi 等，2021）、BrainMRI（Salehi 等，2021）、Br35H（Hamada，2020）以及 COVID-19 检测数据集 COVID-19（Chowdhury 等，2020；Rahman 等，2021）。SOTA 竞争方法包括 CLIP（Radford 等，2021）、CLIP-AC（Radford 等，2021）、WinCLIP（Jeong 等，2023）、VAND（Chen 等，2023）和 CoOp（Zhou 等，2022b）。 我们在附录 A 中提供了关于方法和数据预处理的更多详细信息。异常检测性能通过接收者操作特征曲线下面积（AUROC）进行评估。此外，还使用了异常检测的平均精度（AP）和异常分割的 AUPRO（Bergmann 等，2020）来提供更深入的性能分析。

We use the publicly available CLIP model¹¹1https://github.com/mlfoundations/open_clip
我们使用公开可用的 CLIP 模型 ¹ (VIT-L/14@336px) as our backbone. Model parameters of CLIP are all frozen. The length of learnable word embeddings $E$ is set to 12. The learnable token embeddings are attached to the first 9 layers of the text encoder for refining the textual space, and their length in each layer is set to 4. We fine-tune AnomalyCLIP using the test data on MVTec AD and evaluate the ZSAD performance on other datasets. As for MVTec AD, we fine-tune AomalyCLIP on the test data of VisA. We report dataset-level results, which are averaged across their respective sub-datasets. All experiments are conducted in PyTorch-2.0.0 with a single NVIDIA RTX 3090 24GB GPU. More details can be found in Appendix A.
( VIT-L/14@336px ) 作为我们的骨干网络。CLIP 的模型参数全部冻结。可学习词嵌入 $E$ 的长度设置为 12。可学习的 token 嵌入附加到文本编码器的前 9 层，用于细化文本空间，每层的长度设置为 4。我们使用 MVTec AD 的测试数据对 AnomalyCLIP 进行微调，并在其他数据集上评估 ZSAD 性能。对于 MVTec AD，我们在 VisA 的测试数据上对 AomalyCLIP 进行微调。我们报告数据集级别的结果，这些结果是其各自子数据集的平均值。所有实验均在 PyTorch-2.0.0 中使用单个 NVIDIA RTX 3090 24GB GPU 进行。更多细节可在附录 A 中找到。

Table 1 shows the ZSAD results of AnomalyCLIP with five competing methods over seven industrial defect datasets of very different foreground objects, background, and/or anomaly types. AnomalyCLIP achieves superior ZSAD performance across the datasets, substantially outperforming the other five methods in most datasets. The weak performance of CLIP and CLIP-AC can be attributed to CLIP’s original pre-training, which focuses on aligning object semantics rather than anomaly semantics. By using manually defined text prompts, WinCLIP and VAND achieve better results. Alternatively, CoOp adopts learnable prompts to learn the global anomaly semantics. However, those prompts focus on the global feature and ignore the fine-grained local anomaly semantics, leading to their poor performance on anomaly segmentation. To adapt CLIP to ZSAD, AnomalyCLIP learns object-agnostic text prompts to focus on learning the generic abnormality/normality using global and local context optimization, enabling the modeling of both global and local abnormality/normality. Our resulting prompts can also generalize to different datasets from various domains. To provide more intuitive results, we visualize the anomaly segmentation results of AnomalyCLIP, VAND, and WinCLIP across different datasets in Fig. 4. Compared to VAND and WinCLIP, AnomalyCLIP can perform much more accurate segmentation for the defects from different industrial inspection domains.
表 1 展示了 AnomalyCLIP 与五种竞争方法在七个工业缺陷数据集上的 ZSAD 结果，这些数据集的前景对象、背景和/或异常类型差异很大。AnomalyCLIP 在这些数据集上实现了卓越的 ZSAD 性能，在大多数数据集中显著优于其他五种方法。CLIP 和 CLIP-AC 的弱性能可归因于 CLIP 的原始预训练，其侧重于对齐对象语义而非异常语义。通过使用手动定义的文本提示，WinCLIP 和 VAND 取得了更好的结果。而 CoOp 则采用可学习的提示来学习全局异常语义。然而，这些提示侧重于全局特征，忽略了细粒度的局部异常语义，导致其在异常分割上的表现不佳。为了使 CLIP 适应 ZSAD，AnomalyCLIP 学习与对象无关的文本提示，专注于通过全局和局部上下文优化学习通用的异常/正常性，从而能够建模全局和局部的异常/正常性。我们生成的提示还能够泛化到来自不同领域的不同数据集。 为了提供更直观的结果，我们在图 4 中可视化了 AnomalyCLIP、VAND 和 WinCLIP 在不同数据集上的异常分割结果。与 VAND 和 WinCLIP 相比，AnomalyCLIP 能够对不同工业检测领域的缺陷进行更精确的分割。

To evaluate the generalization ability of our model, we further examine the ZSAD performance of AnomalyCLIP on 10 medical image datasets of different organs across different imaging devices. Table 2 shows the results, where learning-based methods, including AnomalyCLIP, VAND and CoOp, are all tuned using MVTec AD data. It is remarkable that methods like AnomalyCLIP and VAND obtain promising ZSAD performance on various medical image datasets, even though they are tuned using a defect detection dataset. Among all these methods, AnomalyCLIP is the best performer due to its strong generalization brought by object-agnostic prompt learning. As illustrated in Fig. 4, AnomalyCLIP can accurately detect various types of anomalies in diverse medical images, such as skin cancer regions in photography images, colon polyps in endoscopy images, thyroid nodules in ultrasound images, and brain tumors in MRI images, having substantially better performance in locating the abnormal lesion/tumor regions than the other two methods WinCLIP and VAND. This again demonstrates the superior ZSAD performance of AnomalyCLIP in datasets of highly diverse object semantics from medical imaging domains.
为了评估我们模型的泛化能力，我们进一步检查了 AnomalyCLIP 在 10 个不同器官、不同成像设备的医学图像数据集上的 ZSAD 性能。表 2 展示了结果，其中基于学习的方法，包括 AnomalyCLIP、VAND 和 CoOp，均使用 MVTec AD 数据进行调优。值得注意的是，尽管 AnomalyCLIP 和 VAND 等方法使用的是缺陷检测数据集进行调优，但在各种医学图像数据集上仍获得了令人瞩目的 ZSAD 性能。在这些方法中，AnomalyCLIP 表现最佳，这得益于其通过对象无关提示学习带来的强大泛化能力。如图 4 所示，AnomalyCLIP 能够准确检测多种医学图像中的各类异常，如摄影图像中的皮肤癌区域、内窥镜图像中的结肠息肉、超声图像中的甲状腺结节以及 MRI 图像中的脑肿瘤，在定位异常病变/肿瘤区域方面显著优于 WinCLIP 和 VAND 这两种方法。这再次证明了 AnomalyCLIP 在医学成像领域高度多样化的对象语义数据集上卓越的 ZSAD 性能。

Comparing the promising performance in industrial datasets, AnomalyCLIP presents a relatively low performance in medical datasets. This is partly due to the impact of auxiliary data used in our prompt learning. So, then we examine whether the ZSAD performance on medical images can be improved if the prompt learning is trained on an auxiliary medical dataset. One challenge is that there are no available large 2D medical datasets that include both image-level and pixel-level annotations for our training. To address this issue, we create such a dataset based on ColonDB (More details see Appendix A), and then optimize the prompts in AnomalyCLIP and VAND using this dataset and evaluate their performance on the medical image datasets. The results are presented in Table 4. AnomalyCLIP and VAND largely improve their detection and segmentation performance compared to that fine-tuned on MVTec AD, especially for the colon polyp-related datasets such as CVC-ClincDB, Kvasir, and Endo (note that these datasets are all from different domains compared to the fine-tuning ColonDB dataset). AnomalyCLIP also exhibits performance improvement in detecting brain tumors in datasets such as HeadCT, BrainMRI, and Br35H. This is attributed to the visual similarities between colon polyps and brain tumors. Conversely, the symptom of the colon polyp differs significantly from that of diseased skin or chest, leading to performance degradation in ISIC and COVID-19. Overall, compared to VAND, AnomalyCLIP performs consistently better across all datasets of anomaly detection and segmentation.
对比在工业数据集上的优异表现，AnomalyCLIP 在医疗数据集上的表现相对较低。这部分归因于我们提示学习中所用辅助数据的影响。因此，我们随后探讨了若提示学习基于辅助医疗数据集进行训练，是否能提升医疗图像上的零样本异常检测（ZSAD）性能。一个挑战在于，目前尚无包含图像级别和像素级别注释的大型二维医疗数据集可供我们训练使用。为解决这一问题，我们基于 ColonDB 创建了这样一个数据集（更多详情见附录 A），并利用此数据集优化 AnomalyCLIP 和 VAND 中的提示，随后在医疗图像数据集上评估它们的性能。结果如表 4 所示，与在 MVTec AD 上微调相比，AnomalyCLIP 和 VAND 在检测和分割性能上有了显著提升，尤其是在与结肠息肉相关的数据集上，如 CVC-ClincDB、Kvasir 和 Endo（需注意，这些数据集与微调所用的 ColonDB 数据集来自不同领域）。 AnomalyCLIP 在检测如 HeadCT、BrainMRI 和 Br35H 等数据集中的脑肿瘤时也表现出性能提升。这归因于结肠息肉与脑肿瘤之间的视觉相似性。相反，结肠息肉的症状与病变皮肤或胸部的症状差异显著，导致在 ISIC 和 COVID-19 数据集上性能下降。总体而言，与 VAND 相比，AnomalyCLIP 在所有异常检测和分割数据集上均表现出更稳定的优异性能。

To study the effectiveness of object-agnostic prompt learning in AnomalyCLIP, we compare AnomalyCLIP with its variant that uses an object-aware prompt template. The performance gain of AnomalyCLIP to its object-aware prompt learning variant is shown in Fig. 5, where positive values indicate our object-agnostic prompt templates are better than the object-aware one. It is clear that our object-agnostic prompt learning performs much better than, or on par with, the object-aware version in both image-level and pixel-level anomaly detection. This indicates that having object-agnostic prompts helps better learn the generic abnormality and normality in images, as the object semantics are often not helpful, or can even become noisy features, for the ZSAD task.
为了研究 AnomalyCLIP 中对象无关提示学习的有效性，我们将其与使用对象感知提示模板的变体进行了比较。图 5 展示了 AnomalyCLIP 相较于其对象感知提示学习变体的性能提升，正值表明我们的对象无关提示模板优于对象感知模板。显然，在图像级和像素级异常检测中，我们的对象无关提示学习表现远优于或至少与对象感知版本相当。这表明，拥有对象无关提示有助于更好地学习图像中的一般异常和正常情况，因为对象语义对于 ZSAD 任务往往无益，甚至可能成为噪声特征。

We first validate the effectiveness of different high-level modules of our AnomalyCLIP, including DPAM ($T_{1}$), object-agnostic text prompts ($T_{2}$), added learnable tokens in text encoders ($T_{3}$), and multi-layer visual encoder features ($T_{4}$). As shown in Table 4.3, each module contributes to the remarkable performance of AnomalyCLIP. DPAM improves the segmentation performance by enhancing local visual semantics ($T_{1}$). Object-agnostic text prompts focus on the abnormality/normality within images instead of the object semantics, allowing AnomalyCLIP to detect anomalies in diverse unseen objects. Therefore, introducing object-agnostic text prompts ($T_{2}$) significantly improves AnomalyCLIP. Furthermore, text prompt tuning ($T_{3}$) also brings performance improvement via the refinement of original textual space. Finally, $T_{4}$ integrates multi-layer visual semantics to provide more visual details, which further promotes the performance of ZSAD.
我们首先验证了 AnomalyCLIP 中不同高级模块的有效性，包括 DPAM（ $T_{1}$ ）、对象无关的文本提示（ $T_{2}$ ）、文本编码器中添加的可学习标记（ $T_{3}$ ）以及多层视觉编码器特征（ $T_{4}$ ）。如表 4.3 所示，每个模块都对 AnomalyCLIP 的显著性能有所贡献。DPAM 通过增强局部视觉语义（ $T_{1}$ ）提高了分割性能。对象无关的文本提示关注图像中的异常/正常性，而非对象语义，使得 AnomalyCLIP 能够检测各种未见对象的异常。因此，引入对象无关的文本提示（ $T_{2}$ ）显著提升了 AnomalyCLIP。此外，文本提示调优（ $T_{3}$ ）通过优化原始文本空间也带来了性能提升。最后， $T_{4}$ 整合了多层视觉语义以提供更多视觉细节，进一步提升了 ZSAD 的性能。

Next we examine key modules in detail. The object-agnostic prompt learning is the most effective module, and it is driven by our glocal context optimization, so we consider two different optimization terms, local and global losses, in Eq. 2. The results are shown in Table 4.3. Both global and local context optimization contribute to the superiority of AnomalyCLIP. Global context optimization helps to capture global anomaly semantics, thus enabling more accurate image-level detection. Compared to global context optimization, local context optimization incorporates local anomaly semantics, which improves pixel-level performance and complements image-level performance. By synthesizing these two optimization strategies, AnomalyCLIP generally achieves better performance than using them individually.
接下来我们详细检查关键模块。与对象无关的提示学习是最有效的模块，它由我们的全局-局部上下文优化驱动，因此我们在公式 2 中考虑了两种不同的优化项：局部和全局损失。结果如表 4.3 所示。全局和局部上下文优化均对 AnomalyCLIP 的优越性有所贡献。全局上下文优化有助于捕捉全局异常语义，从而实现更准确的图像级检测。与全局上下文优化相比，局部上下文优化融入了局部异常语义，这提升了像素级性能并补充了图像级性能。通过综合这两种优化策略，AnomalyCLIP 通常比单独使用它们时表现更好。

AnomalyCLIP uses $V$-$V$ self-attention by default. Here we study the effectiveness of using two other DPAM strategies, including $Q$-$Q$ and $K$-$K$ self-attention, resulting in two AnomalyCLIP variants, namely AnomalyCLIP${}_{qq}$ and AnomalyCLIP${}_{kk}$. The comparison results are presented in Fig. 6. AnomalyCLIP${}_{qq}$ achieves similar segmentation capabilities as AnomalyCLIP but suffers from degradation in detecting image-level anomalies. Conversely, while AnomalyCLIP${}_{kk}$ performs well in anomaly classification, its segmentation performance is less effective than AnomalyCLIP and AnomalyCLIP${}_{qq}$. The $V$-$V$ self-attention is generally recommended in AnomalyCLIP. Detailed analysis of DPAM can be seen in Appendix C.
AnomalyCLIP 默认使用 $V$ - $V$ 自注意力机制。在此，我们研究了使用另外两种 DPAM 策略的有效性，包括 $Q$ - $Q$ 和 $K$ - $K$ 自注意力机制，从而产生了两个 AnomalyCLIP 变体，即 AnomalyCLIP ${}_{qq}$ 和 AnomalyCLIP ${}_{kk}$ 。比较结果如图 6 所示。AnomalyCLIP ${}_{qq}$ 在分割能力上与 AnomalyCLIP 相当，但在检测图像级异常时表现有所下降。相反，尽管 AnomalyCLIP ${}_{kk}$ 在异常分类上表现良好，但其分割性能不如 AnomalyCLIP 和 AnomalyCLIP ${}_{qq}$ 。在 AnomalyCLIP 中，通常推荐使用 $V$ - $V$ 自注意力机制。DPAM 的详细分析见附录 C。