A Survey on Visual Transformer
Visual Transformer 综述

To handle 2D images, the image $X\in\mathbb{R}^{h\times w\times c}$ is reshaped into a sequence of flattened 2D patches $X_{p}\in\mathbb{R}^{n\times(p^{2}\cdot c)}$ such that $c$ is the number of channels. $(h,w)$ is the resolution of the original image, while $(p,p)$ is the resolution of each image patch. The effective sequence length for the transformer is therefore $n=hw/p^{2}$. Because the transformer uses constant widths in all of its layers, a trainable linear projection maps each vectorized path to the model dimension $d$, the output of which is referred to as patch embeddings.
为了处理2D图像，图像 $X\in\mathbb{R}^{h\times w\times c}$ 被重塑为一系列扁平的2D块 $X_{p}\in\mathbb{R}^{n\times(p^{2}\cdot c)}$ ，使得 $c$ 是通道数。 $(h,w)$ 是原始图像的分辨率，而 $(p,p)$ 是每个图像块的分辨率。因此，转换器的有效序列长度是 $n=hw/p^{2}$ 。因为转换器在其所有层中使用恒定的宽度，可训练的线性投影将每个矢量化路径映射到模型维度 $d$ ，其输出称为补丁嵌入。

Similar to BERT’s $[class]$ token, a learnable embedding is applied to the sequence of embedding patches. The state of this embedding serves as the image representation. During both pre-training and fine-tuning stage, the classification heads are attached to the same size. In addition, 1D position embeddings are added to the patch embeddings in order to retain positional information. It is worth noting that ViT utilizes only the standard transformer’s encoder (except for the place for the layer normalization), whose output precedes an MLP head. In most cases, ViT is pre-trained on large datasets, and then fine-tuned for downstream tasks with smaller data.
类似于BERT的 $[class]$ 令牌，一个可学习的嵌入被应用于嵌入补丁的序列。这种嵌入的状态作为图像表示。在预训练和微调阶段，分类头都被附加到相同的大小。此外，1D位置嵌入被添加到补丁嵌入中，以保留位置信息。值得注意的是，ViT仅利用标准转换器的编码器（除了层归一化的地方），其输出先于MLP头。在大多数情况下，ViT在大型数据集上进行预训练，然后针对数据较小的下游任务进行微调。

ViT yields modest results when trained on mid-sized datasets such as ImageNet, achieving accuracies of a few percentage points below ResNets of comparable size. Because transformers lack some inductive biases inherent to CNNs–such as translation equivariance and locality–they do not generalize well when trained on insufficient amounts of data. However, the authors found that training the models on large datasets (14 million to 300 million images) surpassed inductive bias. When pre-trained at sufficient scale, transformers achieve excellent results on tasks with fewer datapoints. For example, when pre-trained on the JFT-300M dataset, ViT approached or even exceeded state of the art performance on multiple image recognition benchmarks. Specifically, it reached an accuracy of 88.36% on ImageNet, and 77.16% on the VTAB suite of 19 tasks.
当在ImageNet等中型数据集上训练时，ViT产生的结果适中，其准确度比同等大小的ResNets低几个百分点。由于变压器缺乏CNN固有的一些归纳偏差——例如翻译等方差和局部性——当在数据量不足的情况下训练时，它们不能很好地泛化。然而，作者发现，在大型数据集（1400万3亿图像）上训练模型超过了归纳偏差。当在足够规模上进行预训练时，变压器在数据点较少的任务上取得了出色的结果。例如，当在JFT-300M数据集上进行预训练时，ViT在多个图像识别基准上接近甚至超过了最先进的性能。具体而言，它在ImageNet上达到了88.36%的准确率，在VTAB套件（包含19个任务）上达到了77.16%的准确率。

Touvron et al. [60] proposed a competitive convolution-free transformer, called Data-efficient image transformer (DeiT), by training on only the ImageNet database. DeiT-B, the reference vision transformer, has the same architecture as ViT-B and employs 86 million parameters. With a strong data augmentation, DeiT-B achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. In addition, the authors observe that using a CNN teacher gives better performance than using a transformer. Specifically, DeiT-B can achieve top-1 accuracy 84.40% with the help of a token-based distillation.
Touvron等人[60]通过仅在ImageNet数据库上进行训练，提出了一种竞争性的无卷积转换器，称为数据高效图像转换器（DeiT）。参考视觉转换器DeiT-B与ViT-B具有相同的架构，并采用8600万参数。借助强大的数据增强，DeiT-B在没有外部数据的情况下在ImageNet上实现了83.1%（单作物评估）的顶级1准确率。此外，作者观察到使用卷积神经网络教师比使用转换器提供了更好的性能。具体来说，DeiT-B可以在基于令牌的蒸馏的帮助下实现84.40%的顶级1准确率。

The original vision transformer is good at capturing long-range dependencies between patches, but disregard the local feature extraction as the 2D patch is projected to a vector with simple linear layer. Recently, the researchers begin to pay attention to improve the modeling capacity for local information [29, 61, 62]. TNT [29] further divides the patch into a number of sub-patches and introduces a novel transformer-in-transformer architecture which utilizes an inner transformer block to model the relationship between sub-patches and an outer transformer block for patch-level information exchange. Twins [63] and CAT [64] alternately perform local and global attention layer-by-layer. Swin Transformers [61, 65] performs local attention within a window and introduces a shifted window partitioning approach for cross-window connections. Shuffle Transformer [66, 67] further utilizes the spatial shuffle operation instead of shifted window partitioning to allow cross-window connections. RegionViT [62] generates regional tokens and local tokens from an image, and local tokens receive global information via attention with regional tokens. In addition to the local attention, some other works propose to boost local information through local feature aggregation, e.g., T2T [68]. These works demonstrate the benefit of the local information exchange and global information exchange in vision transformer.
最初的视觉转换器擅长捕捉补丁之间的长期依赖关系，但由于2D补丁被投影到具有简单线性层的向量上，因此忽略了局部特征提取。最近，研究人员开始关注提高对局部信息的建模能力[29,61,62]。TNT[29]进一步将补丁划分为多个子补丁，并引入了一种新颖的transformer-in-transformer架构，该架构利用内部转换器块对子补丁和外部转换器块之间的关系进行建模，以进行补丁级信息交换。Twins[63]和CAT[64]交替逐层执行局部和全局注意力。Swin变压器[61,65]在窗口内执行局部注意力，并为跨窗口连接引入了移位窗口分区方法。洗牌Transformer[66,67]进一步利用空间洗牌操作而不是移位窗口分区来允许跨窗口连接。RegionViT[62]从图像中生成区域标记和本地标记，本地标记通过区域标记的注意力接收全局信息。除了局部注意力之外，一些其他工作建议通过局部特征聚合来增强局部信息，例如T2T[68]。这些工作证明了视觉转换器中局部信息交换和全局信息交换的好处。

As a key component of transformer, self-attention layer provides the ability for global interaction between image patches. Improving the calculation of self-attention layer has attracted many researchers. DeepViT [69] proposes to establish cross-head communication to re-generate the attention maps to increase the diversity at different layers. KVT [70] introduces the $k$-NN attention to utilize locality of images patches and ignore noisy tokens by only computing attentions with top-$k$ similar tokens. Refiner [71] explores attention expansion in higher-dimension space and applied convolution to augment local patterns of the attention maps. XCiT [72] performs self-attention calculation across feature channels rather than tokens, which allows efficient processing of high-resolution images. The computation complexity and attention precision of the self-attention mechanism are two key-points for future optimization.
自注意力层作为变压器的关键组成部分，提供了图像块之间全局交互的能力。改进自注意力层的计算吸引了许多研究人员。DeepViT[69]提出建立跨头通信来重新生成注意力图，以增加不同层的多样性。KVT[70]引入了 $k$ -NN注意力，以利用图像块的局部性，并通过仅计算具有top- $k$ 相似标记的注意力来忽略嘈杂的标记。Refiner[71]探索了高维空间中的注意力扩展，并应用卷积来增强注意力图的局部模式。XCiT[72]跨特征通道而不是标记执行自注意力计算，这允许高效处理高分辨率图像。自注意力机制的计算复杂性和注意力精度是未来优化的两个关键点。

The network architecture is an important factor as demonstrated in the field of CNNs. The original architecture of ViT is a simple stack of the same-shape transformer block. New architecture design for vision transformer has been an interesting topic. The pyramid-like architecture is utilized by many vision transformer models [73, 61, 74, 75, 76, 77] including PVT [73], HVT [78], Swin Transformer [61] and PiT [79]. There are also other types of architectures, such as two-stream architecture [80] and U-net architecture [81, 30]. Neural architecture search (NAS) has also been investigated to search for better transformer architectures, e.g., Scaling-ViT [82], ViTAS [83], AutoFormer [84] and GLiT [85]. Currently, both network design and NAS for vision transformer mainly draw on the experience of CNN. In the future, we expect the specific and novel architectures appear in the filed of vision transformer.
网络架构是CNN领域所展示的一个重要因素。ViT的原始架构是相同形状的变压器块的简单堆叠。视觉变压器的新架构设计一直是一个有趣的话题。金字塔状架构被许多视觉变压器模型[73,61,74,75,76,77]使用，包括PVT[73]、HVT[78]、SwinTransformer[61]和PiT[79]。还有其他类型的架构，如双流架构[80]和U-net架构[81,30]。神经架构搜索（NAS）也被研究以搜索更好的变压器架构，例如，缩放-ViT[82]、ViTAS[83]、AutoForm[84]和GLiT[85]。目前，视觉变压器的网络设计和NAS主要借鉴卷积神经网络的经验。未来，我们期待视觉转换器领域出现特定的、新颖的架构。

In addition to the aforementioned approaches, there are some other directions to further improve vision transformer, e.g., positional encoding [86, 87], normalization strategy [88], shortcut connection [89] and removing attention [90, 91, 92, 93].
除了上述方法之外，还有一些其他方向可以进一步改进视觉转换器，例如，位置编码[86,87]、归一化策略[88]、快捷连接[89]和消除注意力[90,91,92,93]。

Although vision transformers have been successfully applied to various visual tasks due to their ability to capture long-range dependencies within the input, there are still gaps in performance between transformers and existing CNNs. One main reason can be the lack of ability to extract local information. Except the above mentioned variants of ViT that enhance the locality, combining the transformer with convolution can be a more straightforward way to introduce the locality into the conventional transformer.
尽管视觉变压器由于能够捕获输入内的远程依赖关系而成功应用于各种视觉任务，但变压器与现有CNN之间在性能方面仍存在差距，一个主要原因可能是缺乏提取局部信息的能力，除了上述增强局部性的ViT变体之外，将变压器与卷积相结合可以是将局部性引入常规变压器的更直接的方式。

There are plenty of works trying to augment a conventional transformer block or self-attention layer with convolution. For example, CPVT [86] proposed a conditional positional encoding (CPE) scheme, which is conditioned on the local neighborhood of input tokens and adaptable to arbitrary input sizes, to leverage convolutions for fine-level feature encoding. CvT [97], CeiT [98], LocalViT [99] and CMT [95] analyzed the potential drawbacks when directly borrowing Transformer architectures from NLP and combined the convolutions with transformers together. Specifically, the feed-forward network (FFN) in each transformer block is combined with a convolutional layer that promotes the correlation among neighboring tokens. LeViT [100] revisited principles from extensive literature on CNNs and applied them to transformers, proposing a hybrid neural network for fast inference image classification. BoTNet [101] replaced the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet, and improved upon the baselines significantly on both instance segmentation and object detection tasks with minimal overhead in latency.
有很多作品试图用卷积来增强传统的变压器块或自注意力层。例如，CPVT[86]提出了一种条件位置编码（CPE）方案，该方案以输入令牌的局部邻域为条件，并可适应任意输入大小，以利用卷积进行精细级别的特征编码。CvT[97]、CeiT[98]、LocalViT[99]和CMT[95]分析了直接从自然语言处理中借用Transformer架构时的潜在缺点，并将卷积与变压器组合在一起。具体来说，每个变压器块中的前馈网络（FFN）与促进相邻令牌之间相关性的卷积层相结合。LeViT[100]重新审视了有关CNN的大量文献中的原理并将其应用于变压器，提出了一种用于快速推理图像分类的混合神经网络。BoTNet[101]在ResNet的最后三个瓶颈块中用全局自关注取代了空间卷积，并以最小的延迟开销显着改进了实例分割和目标检测任务的基线。

Besides, some researchers have demonstrated that transformer based models can be more difficult to enjoy a favorable ability of fitting data [15, 102, 103], in other words, they are sensitive to the choice of optimizer, hyper-parameter, and the schedule of training. Visformer [102] revealed the gap between transformers and CNNs with two different training settings. The first one is the standard setting for CNNs, i.e., the training schedule is shorter and the data augmentation only contains random cropping and horizental flipping. The other one is the training setting used in [60], i.e., the training schedule is longer and the data augmentation is stronger. [103] changed the early visual processing of ViT by replacing its embedding stem with a standard convolutional stem, and found that this change allows ViT to converge faster and enables the use of either AdamW or SGD without a significant drop in accuracy. In addition to these two works, [100, 95] also choose to add convolutional stem on the top of the transformer.
此外，一些研究人员已经证明，基于变压器的模型可能更难享受良好的拟合数据能力[15,102,103]，换句话说，它们对优化器的选择、超参数和训练时间表很敏感。Visger[102]揭示了变压器和CNN之间的差距，两种不同的训练设置。第一个是CNN的标准设置，即训练时间表更短，数据增强只包含随机裁剪和水平翻转。另一个是[60]中使用的训练设置，即训练计划更长，数据增强更强。[103]通过用标准卷积词干替换其嵌入词干来改变ViT的早期视觉处理，并发现这种变化允许ViT更快地收敛，并且能够在精度没有明显下降的情况下使用AdamW或SGD。除了这两个作品之外，[100,95]还选择在变压器顶部添加卷积词干。

We briefly introduce iGPT [14] to demonstrate its mechanism. This approach consists of a pre-training stage followed by a fine-tuning stage. During the pre-training stage, auto-regressive and BERT objectives are explored. To implement pixel prediction, a sequence transformer architecture is adopted instead of language tokens (as used in NLP). Pre-training can be thought of as a favorable initialization or regularizer when used in combination with early stopping. During the fine-tuning stage, they add a small classification head to the model. This helps optimize a classification objective and adapts all weights.
我们简要介绍iGPT[14]来演示其机制。这种方法由预训练阶段和随后的微调阶段组成。在预训练阶段，探索自回归和BERT目标。为了实现像素预测，采用序列转换器架构而不是语言标记（如在自然语言处理中使用的那样）。当与提前终止结合使用时，预训练可以被认为是一个有利的初始化或正则化器。在微调阶段，他们为模型添加了一个小的分类头。这有助于优化分类目标并适应所有权重。

The image pixels are transformed into a sequential data by $k$-means clustering. Given an unlabeled dataset ${X}$ consisting of high dimensional data $\mathbf{x}=(x_{1},\cdots,x_{n})$, they train the model by minimizing the negative log-likelihood of the data:
图像像素通过 $k$ -means聚类转换为顺序数据。给定一个由高维数据 $\mathbf{x}=(x_{1},\cdots,x_{n})$ 组成的未标记数据集 ${X}$ ，它们通过最小化数据的负对数似然来训练模型：

where $p(\mathbf{x})$ is the probability density of the data of images, which can be modeled as:
其中 $p(\mathbf{x})$ 是图像数据的概率密度，可以建模为：

Here, the identity permutation $\pi_{i}=i$ is adopted for $1\leqslant i\leqslant n$, which is also known as raster order. Chen et al. also considered the BERT objective, which samples a sub-sequence $M\subset[1,n]$ such that each index $i$ independently has probability 0.15 of appearing in $M$. $M$ is called the BERT mask, and the model is trained by minimizing the negative log-likelihood of the “masked” elements $x_{M}$ conditioned on the “unmasked” ones $x_{[1,n]\backslash M}$:
这里， $1\leqslant i\leqslant n$ 采用同一性排列 $\pi_{i}=i$ ，也称为光栅顺序。Chen等人还考虑了BERT目标，它对子序列 $M\subset[1,n]$ 进行采样，使得每个索引 $i$ 独立出现在 $M$ 中的概率为0.15。 $M$ 称为BERT掩码，模型通过最小化“掩码”元素 $x_{M}$ 的负对数似然来训练，条件是“未掩码”元素 $x_{[1,n]\backslash M}$ ：

During the pre-training stage, they pick either $L_{AR}$ or $L_{BERT}$ and minimize the loss over the pre-training dataset.
在预训练阶段，他们选择 $L_{AR}$ 或 $L_{BERT}$ 并最小化预训练集的损失。

GPT-2 [110] formulation of the transformer decoder block is used. To ensure proper conditioning when training the AR objective, Chen et al. apply the standard upper triangular mask to the $n\times n$ matrix of attention logits. No attention logit masking is required when the BERT objective is used: Chen et al. zero out the positions after the content embeddings are applied to the input sequence. Following the final transformer layer, they apply a layer norm and learn a projection from the output to logits parameterizing the conditional distributions at each sequence element. When training BERT, they simply ignore the logits at unmasked positions.
使用了转换器解码器块的GPT-2[110]公式。为了确保在训练AR目标时进行适当的调节，Chen等人。将标准的上三角形掩码应用于注意力日志的 $n\times n$ 矩阵。使用BERT目标时不需要注意力日志掩码：Chen等人。将内容嵌入应用于输入序列后的位置归零。在最终的转换器层之后，他们应用层范数并从输出中学习投影，以对每个序列元素的条件分布进行参数化logits。在训练BERT时，他们只需忽略未掩码位置的logits。

During the fine-tuning stage, they average pool the output of the final layer normalization layer across the sequence dimension to extract a $d$-dimensional vector of features per example. They learn a projection from the pooled feature to class logits and use this projection to minimize a cross entropy loss. Practical applications offer empirical evidence that the joint objective of cross entropy loss and pretraining loss ($L_{AR}$ or $L_{BERT}$) works even better. After iGPT, masked image modeling is proposed such as MAE [32] and SimMIM [111] which achieves competitive performance on downstream tasks.
在微调阶段，他们平均池跨序列维度的最终层归一化层的输出，以提取每个示例的 $d$ 维特征向量。他们学习从池化特征到类logits的投影，并使用该投影最小化交叉熵损失。实际应用提供了经验证据，表明交叉熵损失和预训练损失（ $L_{AR}$ 或 $L_{BERT}$ ）的联合目标工作得更好。在iGPT之后，提出了掩蔽图像建模，例如MAE[32]和SimMIM[111]，这在下游任务上实现了竞争性能。

iGPT and ViT are two pioneering works to apply transformer for visual tasks. The difference of iGPT and ViT-like models mainly lies on 3 aspects: 1) The input of iGPT is a sequence of color palettes by clustering pixels, while ViT uniformly divided the image into a number of local patches; 2) The architecture of iGPT is an encoder-decoder framework, while ViT only has transformer encoder; 3) iGPT utilizes auto-regressive self-supervised loss for training, while ViT is trained by supervised image classification task.
iGPT和ViT是将转换器应用于视觉任务的两个开创性工作。iGPT和类ViT模型的区别主要在于3个方面： 1）iGPT的输入是通过聚类像素将调色板序列，而ViT将图像均匀地划分为多个局部块；2）iGPT的架构是编码器-解码器框架，而ViT只有变压器编码器；3）iGPT利用自回归自监督损失进行训练，而ViT是通过监督图像分类任务来训练的。

Chen et al. [31] investigate the effects of several fundamental components for training self-supervised ViT. The authors observe that instability is a major issue that degrades accuracy, and these results are indeed partial failure and they can be improved when training is made more stable.
Chen等人。[31]研究了训练自监督ViT的几个基本组成部分的影响。作者观察到，不稳定性是降低准确性的一个主要问题，这些结果确实是部分失败的，当训练变得更加稳定时，它们可以得到改善。

They introduce a “MoCo v3” framework, which is an incremental improvement of MoCo [114]. Specifically, the authors take two crops for each image under random data augmentation. They are encodes by two encoders, $f_{q}$ and $f_{k}$, with output vectors $\mathbf{q}$ and $\mathbf{k}$. Intuitively, $\mathbf{q}$ behaves like a “query” and the goal of learning is to retrieve the corresponding “key”. This is formulated as minimizing a contrastive loss function, which can be written as:
他们引入了一个“MoCo v3”框架，这是MoCo[114]的增量改进。具体来说，作者在随机数据增强下为每个图像取两个作物。它们由两个编码器编码， $f_{q}$ 和 $f_{k}$ ，输出向量为 $\mathbf{q}$ 和 $\mathbf{k}$ 。直观地说， $\mathbf{q}$ 表现得像一个“查询”，学习的目标是检索相应的“键”。这被表述为最小化一个对比损失函数，可以写成：

Here $\mathbf{k}^{+}$ is $f_{k}$’s output on the same image as $\mathbf{q}$, known as $\mathbf{q}$’s positive sample. The set $\mathbf{k}^{-}$ consists of $f_{k}$’s outputs from other images, known as $\mathbf{q}$’s negative samples. $\tau$ is a temperature hyper-parameter for $l_{2}$-normalized $\mathbf{q}$, $\mathbf{k}$. MoCo v3 uses the keys that naturally co-exist in the same batch and abandon the memory queue, which they find has diminishing gain if the batch is sufficiently large (e.g., 4096). With this simplification, the contrastive loss can be implemented in a simple way. The encoder $f_{q}$ consists of a backbone (e.g., ViT), a projection head and an extra prediction head; while the encoder $f_{k}$ has the backbone and projection head, but not the prediction head. $f_{k}$ is updated by the moving-average of $f_{q}$, excluding the prediction head.
这里 $\mathbf{k}^{+}$ 是 $f_{k}$ 在与 $\mathbf{q}$ 相同的图像上的输出，称为 $\mathbf{q}$ 的正样本。集合 $\mathbf{k}^{-}$ 由 $f_{k}$ 在其他图像上的输出组成，称为 $\mathbf{q}$ 的负样本。 $\tau$ 是 $l_{2}$ -归一化 $\mathbf{q}$ ， $\mathbf{k}$ 的温度超参数。MoCo v3使用在同一批次中自然共存的键，并放弃内存队列，如果批次足够大（例如4096），他们发现该队列的增益会减少。通过这种简化，可以以简单的方式实现对比损失。编码器 $f_{q}$ 由骨干（例如ViT）、投影头和额外的预测头组成；而编码器 $f_{k}$ 具有骨干和投影头，但没有预测头。

MoCo v3 shows that the instability is a major issue of training the self-supervised ViT, thus they describe a simple trick that can improve the stability in various cases of the experiments. They observe that it is not necessary to train the patch projection layer. For the standard ViT patch size, the patch projection matrix is complete or over-complete. And in this case, random projection should be sufficient to preserve the information of the original patches. However, the trick alleviates the issue, but does not solve it. The model can still be unstable if the learning rate is too big and the first layer is unlikely the essential reason for the instability.
MoCo v3表明不稳定性是训练自监督ViT的一个主要问题，因此他们描述了一个简单的技巧，可以在实验的各种情况下提高稳定性。他们观察到没有必要训练补丁投影层。对于标准的ViT补丁大小，补丁投影矩阵是完整的或过完整的。并且在这种情况下，随机投影应该足以保留原始补丁的信息。然而，该技巧缓解了问题，但并没有解决它。如果学习速率太大，并且第一层不太可能是不稳定的本质原因，则模型仍然可能不稳定。

All of the components of vision transformer including multi-head self-attention, multi-layer perceptron, shortcut connection, layer normalization, positional encoding and network topology, play key roles in visual recognition. As stated above, a number of works have been proposed to improve the effectiveness and efficiency of vision transformer. From the results in Figure 6, we can see that combining CNN and transformer achieve the better performance, indicating their complementation to each other through local connection and global connection. Further investigation on backbone networks can lead to the improvement for the whole vision community. As for the self-supervised representation learning for vision transformer, we still need to make effort to pursue the success of large-scale pretraining in the filed of NLP.
视觉转换器的所有组成部分，包括多头自注意、多层感知器、快捷连接、层归一化、位置编码和网络拓扑，在视觉识别中发挥着关键作用。如上所述，已经提出了一些工作来提高视觉转换器的有效性和效率。从图6中的结果可以看出，卷积神经网络和转换器相结合获得了更好的性能，表明它们通过局部连接和全局连接相互补充。对骨干网络的进一步研究可以导致整个视觉社区的改进。至于视觉转换器的自监督表示学习，我们还需要努力追求自然语言处理领域大规模预训练的成功。

Traditional object detectors are mainly built upon CNNs, but transformer-based object detection has gained significant interest recently due to its advantageous capability.
传统的目标检测主要建立在CNN上，但基于变压器的目标检测因其优越的性能而引起了人们的极大兴趣。

Some object detection methods have attempted to use transformer’s self-attention mechanism and then enhance the specific modules for modern detectors, such as feature fusion module [120] and prediction head [121]. We discuss this in the supplemental material. Transformer-based object detection methods are broadly categorized into two groups: transformer-based set prediction methods [16, 17, 122, 123, 124] and transformer-based backbone methods [115, 117], as shown in Fig. 7. Transformer-based methods have shown strong performance compared with CNN-based detectors, in terms of both accuracy and running speed. Table III shows the detection results for different transformer-based object detectors mentioned earlier on the COCO 2012 val set.
一些目标检测方法已经尝试使用变压器的自注意力机制，然后增强现代检测器的特定模块，例如特征融合模块[120]和预测头[121]。我们在补充材料中对此进行了讨论。基于Transformer的目标检测方法大致分为两组：基于变压器的集合预测方法[16,17,122,123,124]和基于变压器的骨干方法[115,117]，如图7所示。与基于卷积神经网络的检测器相比，基于Transformer的方法在精度和运行速度方面表现出了强大的性能。表III显示了前面提到的COCO 2012 val集合上不同基于变压器的目标检测器的检测结果。

DETR is a new design for the object detection framework based on transformer and empowers the community to develop fully end-to-end detectors. However, the vanilla DETR poses several challenges, specifically, longer training schedule and poor performance for small objects. To address these challenges, Zhu et al. [17] proposed Deformable DETR, which has become a popular method that significantly improves the detection performance. The deformable attention module attends to a small set of key positions around a reference point rather than looking at all spatial locations on image feature maps as performed by the original multi-head attention mechanism in transformer. This approach significantly reduces the computational complexity and brings benefits in terms of fast convergence. More importantly, the deformable attention module can be easily applied for fusing multi-scale features. Deformable DETR achieves better performance than DETR with $10\times$ less training cost and $1.6\times$ faster inference speed. And by using an iterative bounding box refinement method and two-stage scheme, Deformable DETR can further improve the detection performance.
DETR是基于变压器的目标检测框架的新设计，并授权社区开发完全端到端的检测器。然而，香草DETR带来了几个挑战，具体来说，较长的训练计划和较差的小对象性能。为了解决这些挑战，朱等人[17]提出了可变形DETR，这已成为一种流行的方法，显着提高了检测性能。可变形注意力模块关注参考点周围的一小部分关键位置，而不是像变压器中的原始多头注意力机制那样查看图像特征图上的所有空间位置。这种方法显着降低了计算复杂性，并在快速收敛方面带来了好处。更重要的是，可变形注意力模块可以轻松应用于融合多尺度特征。Deformable DETR以 $10\times$ 更少的训练成本和 $1.6\times$ 更快的推理速度获得了比DETR更好的性能。并且通过使用迭代包围盒细化方法和两阶段方案，Deformable DETR可以进一步提高检测性能。

There are also several methods to deal with the slow convergence problem of the original DETR. For example, Sun et al. [122] investigated why the DETR model has slow convergence and discovered that this is mainly due to the cross-attention module in the transformer decoder. To address this issue, an encoder-only version of DETR is proposed, achieving considerable improvement in terms of detection accuracy and training convergence. In addition, a new bipartite matching scheme is designed for greater training stability and faster convergence and two transformer-based set prediction models, i.e. TSP-FCOS and TSP-RCNN, are proposed to improve encoder-only DETR with feature pyramids. These new models achieve better performance compared with the original DETR model. Gao et al. [125] proposed the Spatially Modulated Co-Attention (SMCA) mechanism to accelerate the convergence by constraining co-attention responses to be high near initially estimated bounding box locations. By integrating the proposed SMCA module into DETR, similar mAP could be obtained with about 10$\times$ less training epochs under comparable inference cost.
还有几种方法可以处理原始DETR收敛慢的问题。例如，Sun等人[122]调查了为什么DETR模型收敛慢，发现这主要是由于变压器解码器中的交叉注意力模块。为了解决这个问题，提出了一种仅编码器版本的DETR，在检测精度和训练收敛方面实现了相当大的改进。此外，设计了一种新的二分匹配方案，以获得更大的训练稳定性和更快的收敛速度，并提出了两种基于变压器的集合预测模型，即TSP-FCOS和TSP-RCNN，以改进具有特征金字塔的仅编码器DETR。这些新模型与原始DETR模型相比实现了更好的性能。高等人。[125]提出了空间调制协同注意力（SMCA）机制，通过将协同注意力响应限制在靠近初始估计边界盒位置的高水平来加速收敛。通过将所提出的SMCA模块集成到DETR中，在可比推理成本下，可以以大约10 $\times$ 的训练周期获得相似的地图。

Given the high computation complexity associated with DETR, Zheng et al. [123] proposed an Adaptive Clustering Transformer (ACT) to reduce the computation cost of pre-trained DETR. ACT adaptively clusters the query features using a locality sensitivity hashing (LSH) method and broadcasts the attention output to the queries represented by the selected prototypes. ACT is used to replace the self-attention module of the pre-trained DETR model without requiring any re-training. This approach significantly reduces the computational cost while the accuracy slides slightly. The performance drop can be further reduced by utilizing a multi-task knowledge distillation (MTKD) method, which exploits the original transformer to distill the ACT module with a few epochs of fine-tuning. Yao et al.  [126] pointed out that the random initialization in DETR is the main reason for the requirement of multiple decoder layers and slow convergence. To this end, they proposed the Efficient DETR to incorporate the dense prior into the detection pipeline via an additional region proposal network. The better initialization enables them to use only one decoder layers instead of six layers to achieve competitive performance with a more compact network.
鉴于与DETR相关的高计算复杂度，郑等人[123]提出了一种自适应聚类Transformer（ACT），以降低预训练DETR的计算成本。ACT使用局部敏感性散列（LSH）方法自适应地对查询特征进行聚类，并将注意力输出广播到所选原型表示的查询中。ACT用于替换预训练DETR模型的自注意力模块，而不需要任何重新训练。这种方法显着降低了计算成本，同时精度略有下滑。通过利用多任务知识蒸馏（MTKD）方法，可以进一步降低性能下降，该方法利用原始变压器对ACT模块进行几个时期的微调。姚等人[126]指出，DETR中的随机初始化是要求多个解码器层和收敛缓慢的主要原因。为此，他们提出了高效DETR，通过额外的区域提议网络将密集先验合并到检测管道中。更好的初始化使他们能够仅使用一个解码器层而不是六层，以实现与更紧凑网络的竞争性能。

Fang et al. [128] explored how to transfer the pure ViT structure that is pre-trained on ImageNet to the more challenging object detection task and proposed the YOLOS detector. To cope with the object detection task, the proposed YOLOS first drops the classification tokens in ViT and appends learnable detection tokens. Besides, the bipartite matching loss is utilized to perform set prediction for objects. With this simple pre-training scheme on ImageNet dataset, the proposed YOLOS shows competitive performance for object detection on COCO benchmark.
芳等人[128]探索了如何将在ImageNet上预训练的纯ViT结构转移到更具挑战性的目标检测任务中，并提出了YOLOS检测器。为了应对目标检测任务，所提出的YOLOS首先丢弃ViT中的分类标记并附加可学习的检测标记。此外，利用二分匹配损失对对象进行集合预测。通过在ImageNet数据集上这种简单的预训练方案，所提出的YOLOS在COCO基准上显示出对目标检测的竞争力。

Segmentation is an important topic in computer vision community, which broadly includes panoptic segmentation, instance segmentation and semantic segmentation etc. Vision transformer has also shown impressive potential on the field of segmentation.
分割是计算机视觉领域的一个重要课题，广泛地包括全景分割、实例分割和语义分割等。视觉转换器在分割领域也显示出巨大的潜力。

Human pose and hand pose estimation are foundational topics that have attracted significant interest from the research community. Articulated pose estimation is akin to a structured prediction task, aiming to predict the joint coordinates or mesh vertices from input RGB/D images. Here we discuss some methods [35, 36, 37, 119] that explore how to utilize transformer for modeling the global structure information of human poses and hand poses.
人体位姿和手部位姿估计是引起研究界极大兴趣的基础课题。铰接式位姿估计类似于结构化预测任务，旨在从输入RGB/D图像中预测关节坐标或网格顶点。在这里，我们讨论了一些方法[35,36,37,119]，探索如何利用转换器对人体位姿和手部位姿的全局结构信息进行建模。

There are also quite a lot different high/mid-level vision tasks that have explored the usage of vision transformer for better performance. We briefly review several tasks below.
也有相当多不同的高/中级视觉任务探索了视觉转换器的使用以获得更好的性能。我们在下面简要回顾几个任务。

As discussed in the preceding sections, transformers have shown strong performance on several high-level tasks, including detection, segmentation and pose estimation. The key issues that need to be resolved before transformer can be adopted for high-level tasks relate to input embedding, position encoding, and prediction loss. Some methods propose improving the self-attention module from different perspectives, for example, deformable attention [17], adaptive clustering [123] and point transformer [164]. Nevertheless, exploration into the use of transformers for high-level vision tasks is still in the preliminary stages and so further research may prove beneficial. For example, is it necessary to use feature extraction modules such as CNN and PointNet before transformer for potential better performance? How can vision transformer be fully leveraged using large scale pre-training datasets as BERT and GPT-3 do in the NLP field? And is it possible to pre-train a single transformer model and fine-tune it for different downstream tasks with only a few epochs of fine-tuning? How to design more powerful architecture by incorporating prior knowledge of the specific tasks? Several prior works have performed preliminary discussions for the aforementioned topics and We hope more further research effort is conducted into exploring more powerful transformers for high-level vision.
正如前面几节所讨论的，变压器在几个高级任务上表现出了强大的性能，包括检测，分割和姿态估计。在变压器可以用于高级任务之前需要解决的关键问题涉及输入嵌入，位置编码和预测损失。一些方法从不同的角度提出改进自我注意力模块，例如，可变形注意力[17]，自适应聚类[123]和点变压器[164]。尽管如此，对将变压器用于高级视觉任务的探索仍处于初步阶段，因此进一步的研究可能会证明是有益的。例如，是否有必要在变压器之前使用特征提取模块，如卷积神经网络和PointNet，以获得潜在的更好性能？如何使用大规模预训练数据集来充分利用视觉变压器，就像BERT和GPT-3在自然语言处理领域所做的那样？是否有可能预训练单个变压器模型，并仅用几个微调时期就将其精调用于不同的下游任务？如何通过结合特定任务的先验知识来设计更强大的架构？一些先前的工作已经对上述主题进行了初步讨论，我们希望进行更多的进一步研究工作来探索更强大的变压器以实现高级视觉。

An simple yet effective to apply transformer model to the image generation task is to directly change the architectures from CNNs to transformers, as shown in Figure 9 (a). Jiang et al. [39] proposed TransGAN, which build GAN using the transformer architecture. Since the it is difficult to generate high-resolution images pixel-wise, a memory-friendly generator is utilized by gradually increasing the feature map resolution at different stages. Correspondingly, a multi-scale discriminator is designed to handle the varying size of inputs in different stages. Various training recipes are introduced including grid self-attention, data augmentation, relative position encoding and modified normalization to stabilize the training and improve its performance. Experiments on various benchmark datasets demonstrate the effectiveness and potential of the transformer-based GAN model in image generation tasks. Kwonjoon Lee et al. [165] proposed ViTGAN, which introduce several technique to both generator and discriminator to stabilize the training procedure and convergence. Euclidean distance is introduced for the self-attention module to enforce the Lipschitzness of transformer discriminator. Self-modulated layernorm and implicit neural representation are proposed to enhance the training for the generator. As a result, ViTGAN is the first work to demonstrate transformer-based GANs can achieve comparable performance to state-of-the-art CNN-based GANs.
将变压器模型应用于图像生成任务的一个简单但有效的方法是直接将架构从CNN更改为变压器，如图9（a）所示。江等人[39]提出了TransGAN，它使用变压器架构构建GAN。由于很难按像素生成高分辨率图像，因此通过在不同阶段逐渐增加特征图分辨率来利用内存友好的生成器。相应地，设计了一个多尺度鉴别器来处理不同阶段输入的不同大小。引入了各种训练食谱，包括网格自注意、数据增强、相对位置编码和修改的归一化，以稳定训练并提高其性能。在各种基准数据集上的实验证明了基于变压器的GAN模型在图像生成任务中的有效性和潜力。[165]提出了ViTGAN，它在生成器和鉴别器中引入了几种技术来稳定训练过程和收敛。自我注意力模块引入了欧几里德距离来增强变压器鉴别器的Lipschitzness。提出了自调制层范数和隐式神经表示来增强生成器的训练。因此，ViTGAN是第一个证明基于变压器的GAN可以实现与最先进的基于卷积神经网络的GAN相当的性能的工作。

Parmar et al. [27] proposed Image Transformer, taking the first step toward generalizing the transformer model to formulate image translation and generation tasks in an auto-regressive manner. Image Transformer consists of two parts: an encoder for extracting image representation and a decoder to generate pixels. For each pixel with value $0-255$, a $256\times d$ dimensional embedding is learned for encoding each value into a $d$ dimensional vector, which is fed into the encoder as input. The encoder and decoder adopt the same architecture as that in [9]. Each output pixel $q^{\prime}$ is generated by calculating self-attention between the input pixel $q$ and previously generated pixels $m_{1},m_{2},...$ with position embedding $p_{1},p_{2},...$. For image-conditioned generation, such as super-resolution and inpainting, an encoder-decoder architecture is used, where the encoder’s input is the low-resolution or corrupted images. For unconditional and class-conditional generation (i.e., noise to image), only the decoder is used for inputting noise vectors. Because the decoder’s input is the previously generated pixels (involving high computation cost when producing high-resolution images), a local self-attention scheme is proposed. This scheme uses only the closest generated pixels as input for the decoder, enabling Image Transformer to achieve performance on par with CNN-based models for image generation and translation tasks, demonstrating the effectiveness of transformer-based models on low-level vision tasks.
Parmar等人[27]提出了图像Transformer，朝着泛化变压器模型迈出了第一步，以自回归的方式制定图像翻译和生成任务。图像Transformer由两部分组成：用于提取图像表示的编码器和用于生成像素的解码器。对于每个具有值 $0-255$ 的像素，学习了 $256\times d$ 维度嵌入，用于将每个值编码成 $d$ 维度向量，该向量作为输入馈入编码器。编码器和解码器采用与[9]中相同的架构。每个输出像素 $q^{\prime}$ 是通过计算输入像素 $q$ 和先前生成的像素 $m_{1},m_{2},...$ 之间的自关注来生成的，具有位置嵌入 $p_{1},p_{2},...$ 。对于图像条件生成，如超分辨率和修复，使用编码器-解码器架构，其中编码器的输入是低分辨率或损坏的图像。对于无条件和类条件生成（即噪声到图像），仅使用解码器输入噪声向量。由于解码器的输入是先前生成的像素（在生成高分辨率图像时涉及高计算成本），因此提出了一种局部自注意方案。该方案仅使用最接近的生成像素作为解码器的输入，使图像Transformer能够在图像生成和翻译任务方面实现与基于卷积神经网络的模型相当的性能，证明了基于变压器的模型在低级视觉任务上的有效性。