Hyperspectral Image Classification Using 3D Attention Mechanism in Collaboration with Transformer
基于 3D 注意力机制与 Transformer 协作的高光谱图像分类

Hyperspectral images not only exhibit high dimensionality in spectral dimension, but also correlation exists between different bands of data and high information redundancy. In order to eliminate the redundancy between spectra, Principal Component Analysis (PCA) is used to reduce the number of spectral bands, and the spatial information is retained by reducing the dimensionality while still maintaining the original spatial dimensions. Firstly, the feature maps are cut, and the dimensionality reduction operation is performed on the data set using PCA, and the reduced feature maps are put into the network.
高光谱图像不仅在光谱维度上表现出高维度，而且不同波段的数据之间存在相关性以及高信息冗余。为了消除光谱之间的冗余，使用主成分分析（PCA）来减少光谱波段的数量，并通过降维同时保持原始空间维度来保留空间信息。首先，对特征图进行裁剪，然后使用 PCA 对数据集进行降维操作，将降维后的特征图输入网络。

The main idea of PCA is to map n-dimensional features to k-dimensions, which are new orthogonal features also called principal components, reconstructing k-dimensional features based on the original n-dimensional features. PCA works by sequentially finding a set of mutually orthogonal axes from the original space, and the selection of new axes is closely related to the data itself. The first new axis is chosen in the direction of the largest variance in the original data, the second new axis is chosen in the plane orthogonal to the first axis that makes the largest variance, and the third axis is in the plane orthogonal to the 1st and 2nd axes that makes the largest variance. By analogy, n such axes can be obtained. The new axes are obtained in this way. Most of the variance is contained in the first k axes, and the latter axes contain almost zero variance, so that the remaining axes are ignored and only the first k axes containing most of the variance are retained. In fact, this is equivalent to keeping only the dimensional features that contain most of the variance, and ignoring the dimensional features that contain almost zero variance, to realize the dimensionality reduction of the data features.
PCA 的主要思想是将 n 维特征映射到 k 维，这些新的正交特征也称为主成分，并基于原始的 n 维特征重构 k 维特征。PCA 通过从原始空间中依次找到一组相互正交的轴来工作，新轴的选择与数据本身密切相关。第一个新轴选择在原始数据方差最大的方向上，第二个新轴选择在与第一个轴正交的平面上方差最大的方向上，第三个轴则选择在与第 1 和第 2 个轴正交的平面上方差最大的方向上。依此类推，可以获得 n 个这样的轴。新轴就是这样获得的。大部分方差包含在前 k 个轴中，而后面的轴几乎不包含方差，因此可以忽略剩余的轴，只保留包含大部分方差的前 k 个轴。 实际上，这相当于只保留包含大部分方差的维度特征，而忽略掉几乎零方差的维度特征，从而实现数据特征的降维。

Take the Indian Pines dataset as an example, its size is 145 × 145 × 200, where 145 × 145 denotes the height and width of the image, respectively, and 200 denotes the spectral dimension. Assuming that the original input image size is M × N × C, M is the height of the image, N is the width of the image, and C is the number of bands of the spectrum, and the first B principal components are retained by using PCA, then the image size can be changed to M × N × B by PCA dimensionality reduction, which eliminates the redundancy of the spectrum while retaining the spatial information.
以 Indian Pines 数据集为例，其大小为 145 × 145 × 200，其中 145 × 145 分别表示图像的高度和宽度，200 表示光谱维度。假设原始输入图像的大小为 M × N × C，M 是图像的高度，N 是图像的宽度，C 是光谱的波段数，并且通过 PCA 保留了前 B 个主成分，则可以通过 PCA 降维将图像大小变为 M × N × B，这样既消除了光谱的冗余，又保留了空间信息。

Assuming that all pixels in the p-neighborhood of each pixel point are selected as a sample, the sample after neighborhood extraction can be expressed as:
假设每个像素点的 p 邻域内的所有像素都被选为样本，那么在邻域提取后的样本可以表示为：

When a pixel point is located at the edge of an image, its neighborhood will exceed the original size of the image, and the excess is filled with 0. The label of the central pixel point is used as the true label of this sample. Let K= 2p+ 1, the size of each sample becomes K × K × B. After neighborhood extraction, the 3D stereo hyperspectral image data is divided into small overlapping 3D image blocks (Patch), and the spectral dimension of each patch is still the size after dimensionality reduction, only the spatial size has changed.
当一个像素点位于图像边缘时，其邻域将超出原始图像的大小，超出部分用 0 填充。中心像素点的标签被用作该样本的真实标签。设 K=2p+1，每个样本的大小变为 K × K × B。经过邻域提取后，三维立体高光谱图像数据被分割成小的重叠三维图像块（Patch），每个 Patch 的光谱维度仍然是降维后的大小，只有空间大小发生了变化。

The final divided data is input to the network in the form of 9*9*30. Firstly, a convolutional kernel of 3*3*C is used in the tandem 3D-CNN, where C decreases with the deepening of the network, and separate space and separate channels are taken for the 3D-CNN used for SAM and CAM. The parameters used in the specific network regarding the channel attention mechanism and the spatial attention mechanism are shown in Table 1.Subsequently, the 2D-CNN is added after the 3D-CNN to enable the network to acquire the spatial information of the feature map again. The use of a separate layer of 2D-CNN ensures that the convolution will not be overfitted, but also ensures that the network can learn the spatial feature information of the hyperspectral image.
最终划分的数据以 9*9*30 的形式输入到网络中。首先，在串联的 3D-CNN 中使用 3*3*C 的卷积核，其中 C 随着网络的加深而减少，并且为用于 SAM 和 CAM 的 3D-CNN 分别采用空间和通道分离的方式。关于通道注意力机制和空间注意力机制的具体网络参数如表 1 所示。随后，在 3D-CNN 之后添加 2D-CNN，使网络能够再次获取特征图的空间信息。使用单独一层 2D-CNN 确保卷积不会过拟合，同时也确保网络可以学习高光谱图像的空间特征信息。

Spatial Attention Mechanisms (SAM) aims to enhance the feature representation of key regions, essentially transforming the spatial information in the original image into another space and preserving the key information through a spatial transformation module, generating a weighted mask for each location and weighting the output so as to enhance specific target regions of interest while weakening irrelevant background regions.
空间注意力机制（SAM）旨在增强关键区域的特征表示，本质上是将原始图像中的空间信息转换到另一个空间，并通过空间变换模块保留关键信息，为每个位置生成加权掩码，并对输出进行加权，以增强特定的目标感兴趣区域，同时削弱无关的背景区域。

Channel Attention Mechanisms (CAM) is designed to model the correlation between different channels and automatically obtain the importance of each feature channel through network learning, and then assign different weight factors to each channel to reinforce the important features and suppress the non-important ones.
通道注意力机制（CAM）旨在建模不同通道之间的相关性，并通过网络学习自动获得每个特征通道的重要性，然后为每个通道分配不同的权重因子以增强重要特征并抑制不重要的特征。

The 3D-CNN and 2D-CNN alone do not provide good classification accuracy in the process of processing hyperspectral images with little acquired data. Although 3D-CNN can directly acquire the null spectrum information in image features at the same time, the acquired feature information cannot know the internal situation of the feature map. To address the above situation, SAM and CAM are introduced into 3D-CNN, using CAM can make the network focus on more discriminative channels while suppressing unnecessary channel information, and similarly, SAM can make the network focus more on spatial texture information. In this paper, we compare the original spatial attention mechanism with the traditional channel attention mechanism, and make an innovation. The comparison diagram is shown in Figure 2.
3D-CNN 和 2D-CNN 在处理数据量较少的高光谱图像时，单独使用并不能提供良好的分类精度。尽管 3D-CNN 可以直接获取图像特征中的空谱信息，但所获取的特征信息无法了解特征图内部的情况。为了解决上述情况，将 SAM 和 CAM 引入到 3D-CNN 中，使用 CAM 可以使网络更关注更具区分性的通道，同时抑制不必要的通道信息，同样地，SAM 可以使网络更关注空间纹理信息。在本文中，我们将原始的空间注意力机制与传统的通道注意力机制进行了比较，并进行了创新。比较图如图 2 所示。

Figure 2: SAM/CAM of variant residual structure

The single channel attention mechanism branch, which contains only 1∼2 convolutions, is lacking for channel and space learning compared to using multiple convolutions, but due to the internal algorithm of convolution itself, the more convolutions are used for image learning, the easier it is to cause overfitting phenomenon, so this paper finally uses 3 3D-CNNs for channel and space learning of feature maps, and adds BN layer after each 3D-CNN to prevent over-learning and under-learning of feature information.
单通道注意力机制分支仅包含 1~2 个卷积，与使用多个卷积相比，在通道和空间学习方面存在不足，但由于卷积本身的内部算法，用于图像学习的卷积越多，越容易导致过拟合现象。因此，本文最终使用 3 个 3D-CNN 进行特征图的通道和空间学习，并在每个 3D-CNN 后添加 BN 层，以防止特征信息的过学习和欠学习。

The conventional Transformer processes image features in the form of a one-dimensional sequence. To process 2D images, the conventional Transformer reshapes the image${\rm{x}} \in {\mathbb{R}}^{H \times W \times C}$into a sequence of flat 2D blocks ${{\rm{x}}}_P \in {\mathbb{R}}^{N \times ({p}^2\centerdot C)}$, where (H,W) is the resolution of the original image, C is the number of channels, (P,P) is the resolution of each image block and$N = HW/{p}^2$is the number of blocks obtained, and N will also be used as the effective input sequence length of the Transformer. The Transformer in the hidden layer uses a constant potential vector size D, patch is stretched and mapped to the D dimension using a trainable linear projection (Eq. 2). The output of this projection is called patch embedding in this paper.
传统的 Transformer 以一维序列的形式处理图像特征。为了处理二维图像，传统的 Transformer 将图像${\rm{x}} \in {\mathbb{R}}^{H \times W \times C}$重塑为一系列平坦的二维块${{\rm{x}}}_P \in {\mathbb{R}}^{N \times ({p}^2\centerdot C)}$，其中(H, W)是原始图像的分辨率，C 是通道数，(P, P)是每个图像块的分辨率，$N = HW/{p}^2$是获得的块数，N 也将用作 Transformer 的有效输入序列长度。隐藏层中的 Transformer 使用恒定的潜在向量大小 D，通过可训练的线性投影（公式 2）将块拉伸并映射到 D 维。这个投影的输出在本文中称为块嵌入。

The Transformer adds a learnable embedding before the embedded patch sequence ($Z_0^0 = {X}_{class}$) with the state at the output of its encoder ($Z_L^0$) as the image representation y (Eq. 5). The cls are appended to $Z_L^0$ during pre-training. The cls are implemented by an MLP in an implicit layer during pre-training and by a linear layer in the FFN layer. The position information of the feature map is added to the patch embeding to preserve the position information. Transformer adds cls to the feature sequence as input to the embedding vector sequence setting encoder.
Transformer 在嵌入的 patch 序列之前添加了一个可学习的嵌入（$Z_0^0 = {X}_{class}$），其编码器输出的状态（$Z_L^0$）作为图像表示 y（公式 5）。在预训练期间，cls 被附加到$Z_L^0$。在预训练期间，cls 通过一个隐层中的 MLP 实现，在 FFN 层中则通过一个线性层实现。特征图的位置信息被添加到 patch 嵌入中以保留位置信息。Transformer 将 cls 添加到特征序列中，作为嵌入向量序列设置编码器的输入。

The Transformer encoder consists of an MLP block FFN layer (Eq. 3,4) and a Multihead Self-Attention (MSA)[12] . Layernorm (LN) is applied before MSA and FFN layers, and residual connections are applied after each block[13].
Transformer 编码器由一个 MLP 块 FFN 层（公式 3,4）和一个多头自注意力机制（MSA）[12]组成。在 MSA 和 FFN 层之前应用了层归一化（LN），并在每个块之后应用了残差连接[13]。

The MLP contains two layers with GELU nonlinearity.
MLP 包含两层，使用 GELU 非线性激活函数。

Self-attention contains three elements, query Q, key K, and value V. For each element in the input sequence $z \in {\mathbb{R}}^{N \times D}$, we compute the weighted sum of all values v in the sequence. The attention weight ${A}_{ij}$ is based on the pairwise similarity between two elements of the sequence and their respective query ${q}^{\rm{i}}$ and key ${k}^j$ representations.
自注意力机制包含三个元素，查询 Q、键 K 和值 V。对于输入序列 $z \in {\mathbb{R}}^{N \times D}$ 中的每个元素，我们计算序列中所有值 v 的加权和。注意力权重 ${A}_{ij}$ 基于序列中两个元素及其各自的查询 ${q}^{\rm{i}}$ 和键 ${k}^j$ 表示之间的成对相似性。

Multi-headed self-attentiveness (MSA) is an extension of SA that runs k self-attentive operations, called "heads", in parallel and projects their connected outputs. To keep the amount of computation and the number of parameters constant when changing k, ${D}_h$ (Eq. 7) is usually set to D/k.
多头自注意力（MSA）是自注意力（SA）的扩展，它并行运行 k 个自注意力操作，称为“头”，并将它们的输出连接起来。为了在改变 k 时保持计算量和参数数量不变，${D}_h$（公式 7）通常设置为 D/k。

For the Transformer model, in order to make the Transformer have better learning ability, this paper is improved for the information input to the VIT module, and the sequences are passed into the Transformer. It seems that the neighboring sequence blocks may have similar characteristics in the natural image, but relative to the hyperspectral image, because of its image characteristics at the neighboring positions of the null spectrum The information contained in the features is different, the sequence block and the neighboring sequence blocks do not closely understand their correlation, so this paper adopts the group transfer method to gradually pass in the feature sequences into Transformer, the group transfer method is shown in Figure 3.
对于 Transformer 模型，为了使 Transformer 具有更好的学习能力，本文改进了输入到 VIT 模块的信息，并将序列传递到 Transformer 中。在自然图像中，相邻的序列块可能具有相似的特征，但相对于高光谱图像而言，由于其在零谱相邻位置的图像特性所包含的信息不同，序列块与其相邻序列块之间的关联性并不紧密，因此本文采用分组传递方法逐步将特征序列传递到 Transformer 中，分组传递方法如图 3 所示。

Figure 3: Grouped transfer structure diagram

In this paper, four different adjacent blocks are passed into the Transformer at one time, and the subsequent sequences are passed in a progressive manner, moving one sequence block at a time to ensure that in the Transformer model, each adjacent block can be learned. Given a spectral feature (a pixel in the HS image) $y = [{y}_1,{y}_2, \cdots ,{y}_m] \in {\mathbb{R}}^{1 \times m}$, the feature embeddings a obtained by classic transformers are formulated by
在本文中，四个不同的相邻块一次性输入到 Transformer 中，随后的序列以递进的方式输入，每次移动一个序列块，以确保在 Transformer 模型中，每个相邻块都可以被学习。给定一个光谱特征（HS 图像中的一个像素）$y = [{y}_1,{y}_2, \cdots ,{y}_m] \in {\mathbb{R}}^{1 \times m}$，通过经典 Transformer 获得的特征嵌入表示为

All experiments in this paper were run on 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz, NVIDIA GeForce RTX 3060, 16G RAM hardware support. The implementation is based on Windows 10 using the Pytorch framework with Python 3.6.14.
本文中的所有实验均在 11 代 Intel(R) Core(TM) i7-11800H @ 2.30GHz、NVIDIA GeForce RTX 3060、16G RAM 硬件支持下运行。实现基于 Windows 10 操作系统，使用 Pytorch 框架和 Python 3.6.14。

To verify the effectiveness of the method in this paper, experiments are conducted on 2 publicly available datasets of hyperspectral images, namely Indian Pines (IP) and Pavia University (PU).
为了验证本文方法的有效性，在两个公开的高光谱图像数据集上进行了实验，即 Indian Pines (IP) 和 Pavia University (PU)。

The IP dataset is the earliest test data for hyperspectral image classification, imaged by the Airborne Visual Infrared Imaging Spectrometer (AVIRIS) in 1992 on an Indian pine tree in Indiana, USA, and then intercepted to a size of 145×145 for annotation as hyperspectral image classification test purpose. The spatial resolution is 17m/pixel, with 200 channel bands from 400nm to 2500nm, and 24 noise bands removed. It can classify 10249 pixels, including 16 kinds of features such as forest, corn, oats and soybean. Due to the insufficient amount of data, the feature boundaries are not clear and the feature boundaries in the IP dataset cannot be mapped accurately.
IP 数据集是最早的高光谱图像分类测试数据，由机载视觉红外成像光谱仪（AVIRIS）于 1992 年在美国印第安纳州的一棵印第安松树上成像，然后截取为 145×145 的大小，用于标注作为高光谱图像分类测试目的。空间分辨率为 17 米/像素，包含从 400 纳米到 2500 纳米的 200 个通道波段，并去除了 24 个噪声波段。它可以分类 10249 个像素，包括森林、玉米、燕麦和大豆等 16 种特征。由于数据量不足，特征边界不清晰，IP 数据集中的特征边界无法准确映射。

The PU dataset was collected using a spectral imager at the Pavia University, Italy, with a spatial size of 610 × 340 pixels and a spatial resolution of 1.3 m/pixel, possessing 103 channel bands between 430 nm and 860 nm, with 42,776 classifiable image elements, containing nine feature classes such as woods, bricks, and gravels. The high spatial resolution of scattering-prone objects, such as woods and sidewalks, poses great difficulties for feature learning based on CNNs.
PU 数据集是在意大利帕维亚大学使用光谱成像仪收集的，空间大小为 610×340 像素，空间分辨率为 1.3 米/像素，拥有 430 纳米到 860 纳米之间的 103 个通道波段，包含 42,776 个可分类的图像元素，包含九种特征类别，如树林、砖块和碎石。易散射物体（如树林和人行道）的高空间分辨率给基于 CNN 的特征学习带来了很大的困难。

The evaluation criteria use Overall Accuracy (OA), Average Accuracy (AA) and Kappa (KA) coefficients to determine the classification performance of the model.
评估标准使用总体准确率（OA）、平均准确率（AA）和 Kappa（KA）系数来确定模型的分类性能。

10% of the IP data set is extracted as the training set, and the number of pixels per pixel point is shown in the following table.
10%的 IP 数据集被提取作为训练集，每个像素点的像素数量如下表所示。

5% of the PU data set is extracted as the training set, and the number of pixels per pixel point is shown in the following table.
5%的 PU 数据集被提取作为训练集，每个像素点的像素数量如下表所示。

Several common network models, 2D-CNN, ContextualNet, PyResNet, ResNet, SPRVIT, and Transformer, were selected for comparative analysis and used to verify the effectiveness of the proposed method. To ensure the fairness of the experimental results, each experiment is conducted in the same environment and each parameter setting is the same. In order to verify the advantages of 3D-AMSTN, its classification performance is compared with several network models mentioned above, where the settings of each parameter and the input size are the same. The experimental results on the 2 datasets, IP and PU, are listed in Table 3 and Table 4.
选取了几种常见的网络模型，包括 2D-CNN、ContextualNet、PyResNet、ResNet、SPRVIT 和 Transformer，进行对比分析，并用于验证所提出方法的有效性。为了确保实验结果的公平性，每个实验都在相同的环境中进行，且每个参数设置都相同。为了验证 3D-AMSTN 的优势，将其分类性能与上述几种网络模型进行了比较，其中每个参数的设置和输入大小都相同。在 IP 和 PU 两个数据集上的实验结果列于表 3 和表 4 中。

From the overall experimental results on the 2 datasets, it is clear that the 3D-AMSTN has a high classification accuracy with an overall accuracy of more than 99% on both datasets. On the IP dataset, the proposed 3D-AMSTN is 39.7% higher in OA, 41.57% higher in AA, and 45.98% higher in Kappa compared with the traditional Transformer network model, which is due to the insufficient local information acquisition capability of the traditional Transformer, and this paper improves the information input to the VIT module by making the Transformer has better learning ability. Compared with 2D-CNN based on spatial extraction, OA is 8.17% higher, AA is 10.23% higher, and Kappa is 9.35% higher. This is due to the fact that CNN can only acquire hyperspectral image features in local context, and the variant Transformer model proposed in this paper enables the network to obtain better global feature information on feature sequences, thus improving the classification accuracy. The classification accuracy is also improved by 0.89% for OA, 1.65% for AA, and 1.01% for Kappa compared with SPRVIT, which has the best performance among several other classification models. On the PU dataset, the 3D-AMSTN proposed in this paper also performs well, with OA, AA, and Kappa improved by 12.16%, 16.6%, and 16.21%, respectively, compared with the 2D-CNN network model based on spatial extraction. Compared with the best-performing SPRVIT, OA is 2.89% higher, AA is 6.29% higher, and Kappa is 3.86% higher. The proposed 3D-AMSTN outperforms several other classification models in all performance metrics, which fully demonstrates the effectiveness of 3D-AMSTN in improving classification accuracy.
从 2 个数据集的整体实验结果来看，3D-AMSTN 具有很高的分类精度，在两个数据集上的总体精度都超过 99%。在 IP 数据集上，所提出的 3D-AMSTN 相比传统的 Transformer 网络模型，OA 提高了 39.7%，AA 提高了 41.57%，Kappa 提高了 45.98%，这是由于传统 Transformer 获取局部信息的能力不足，本文通过改进输入到 VIT 模块的信息，使 Transformer 具有更好的学习能力。与基于空间提取的 2D-CNN 相比，OA 提高了 8.17%，AA 提高了 10.23%，Kappa 提高了 9.35%。这是因为 CNN 只能在局部上下文中获取高光谱图像特征，而本文提出的变体 Transformer 模型使网络能够在特征序列中获得更好的全局特征信息，从而提高了分类精度。与表现最好的其他几种分类模型中的 SPRVIT 相比，OA 提高了 0.89%，AA 提高了 1.65%，Kappa 提高了 1.01%。 在 PU 数据集上，本文提出的 3D-AMSTN 也表现良好，与基于空间提取的 2D-CNN 网络模型相比，OA、AA 和 Kappa 分别提高了 12.16%、16.6%和 16.21%。与表现最好的 SPRVIT 相比，OA 高出 2.89%，AA 高出 6.29%，Kappa 高出 3.86%。所提出的 3D-AMSTN 在所有性能指标上都优于其他几种分类模型，充分证明了 3D-AMSTN 在提高分类精度方面的有效性。

Figure 4 and Figure 5 give the classification results of this paper's method and the comparison method under the IP and PU datasets, respectively. It is obvious from Fig. 4 and Fig. 5 that the proposed method in this paper has fewer misclassification points. The 2D-CNN and Transformer-based classification methods differ more from the reference samples, and the classification results are much inferior to those of the proposed method. The rest of the classification methods have improved the classification effect, but the classification effect is slightly inferior to that of the proposed method.
图 4 和图 5 分别展示了本文方法与对比方法在 IP 和 PU 数据集上的分类结果。从图 4 和图 5 可以看出，本文提出的方法误分类点较少。基于 2D-CNN 和 Transformer 的分类方法与参考样本差异较大，分类结果远不如本文提出的方法。其余分类方法的分类效果有所改善，但仍然略逊于本文提出的方法。

Figure 4: Classification results of different models in IP dataset

Figure 5 Classification results of different models in PU dataset

In addition, the classification results of the method in this paper are comparable to other deep learning methods when the features of the features in the images are easier to distinguish. For example, the Oats and Grass-trees classes in the IP dataset and the Metal sheets and Shadow classes in the PU dataset are easy to distinguish, and they can achieve good classification results. The classification accuracy of the method in this paper is also improved for feature types with similar features that are prone to errors in classification.
此外，当图像中的特征更容易区分时，本文方法的分类结果与其他深度学习方法相当。例如，在 IP 数据集中，燕麦和草树类别以及在 PU 数据集中的金属板和阴影类别都容易区分，并且可以实现良好的分类结果。对于容易在分类中出错的相似特征类型，本文方法的分类精度也得到了提高。

In order to verify the rationality of each module of 3D-AMSTN, the effectiveness of setting different modules is experimentally demonstrated with the IP dataset as an example, and the comparison results are shown in Table 5.
为了验证 3D-AMSTN 每个模块的合理性，实验以 IP 数据集为例展示了设置不同模块的有效性，比较结果如表 5 所示。

As shown in the above table, it can be seen that gradually adding channel attention mechanism and channel attention mechanism has a significant increase for the classification of hyperspectral, and finally adding Transformer can find that the network can classify better on hyperspectral images in terms of global information, and the three methods collaborate with each other to make the classification accuracy of the network reach more than 99%.
如上表所示，可以发现逐步添加通道注意力机制和通道注意力机制对高光谱图像分类有显著提升，最终添加 Transformer 后可以发现网络在全局信息方面能够更好地对高光谱图像进行分类，三种方法相互协作使网络的分类准确率达到 99%以上。