Attention Is All You Need
注意力就是你所需要的一切 (Attention Is All You Need)

Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35, 2, 5]. Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].
循环神经网络（Recurrent neural networks），特别是长短期记忆网络（long short-term memory [13]）和门控循环神经网络（gated recurrent [7]），已被确立为序列建模和转换问题（如语言建模和机器翻译 [35, 2, 5]）中的最先进方法（state of the art approaches）。此后，许多努力持续推动循环语言模型和编码器-解码器架构（encoder-decoder architectures [38, 24, 15]）的界限。

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states $h_{t}$, as a function of the previous hidden state $h_{t-1}$ and the input for position $t$. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.
循环模型通常会根据输入和输出序列的符号位置来分解计算。将这些位置与计算时间的步骤对齐，它们会生成一系列隐藏状态 $h_{t}$ ，作为前一个隐藏状态 $h_{t-1}$ 和位置 $t$ 的输入的函数。这种本质上的顺序性质排除了在训练样本内的并行化，这在序列长度较长时变得至关重要，因为内存限制限制了跨样本的批处理。最近的工作通过分解技巧 [21] 和条件计算 [32] 在计算效率上取得了显著的改进，同时在后者的情况下也提高了模型性能。然而，顺序计算的基本约束仍然存在。 (循环模型 Recurrent models: 一种处理序列数据的神经网络模型) (符号位置 symbol positions: 在序列中的每个元素的位置) (隐藏状态 hidden states: 神经网络中不直接输出但用于存储信息的状态) (分解计算 factor computation: 将复杂计算分解为更简单的步骤) (并行化 parallelization: 同时进行多个计算以提高效率) (内存限制 memory constraints: 计算机内存的限制) (批处理 batching: 将多个样本一起处理以提高效率) (分解技巧 factorization tricks: 通过将问题分解为更小的部分来简化计算) (条件计算 conditional computation: 根据输入的不同条件来决定计算路径)

Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences [2, 19]. In all but a few cases [27], however, such attention mechanisms are used in conjunction with a recurrent network.
注意机制已经成为各种任务中引人注目的序列建模和转换模型不可或缺的一部分，使得建模依赖关系时无需考虑输入或输出序列中距离的远近 [2, 19]。然而，除了少数情况 [27] 之外，这些注意机制通常与循环网络结合使用。 (注意机制 Attention mechanisms) (序列建模 sequence modeling) (转换模型 transduction models) (依赖关系 dependencies) (循环网络 recurrent network)

In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.
在本研究中，我们提出了 Transformer，一种模型架构，它摒弃了递归机制，完全依赖于注意力机制来捕捉输入和输出之间的全局依赖关系。Transformer 允许显著更多的并行化处理，并且在八个 P100 GPU 上训练仅需十二小时即可达到翻译质量的全新最先进水平（state of the art）。(P100 GPU: 英伟达的一种图形处理单元)

The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building block, computing hidden representations in parallel for all input and output positions. In these models, the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes it more difficult to learn dependencies between distant positions [12]. In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as described in section 3.2.
减少顺序计算的目标也是扩展神经 GPU [ 16]、ByteNet [ 18] 和 ConvS2S [ 9] 的基础，这些模型都使用卷积神经网络作为基本构建模块，并行计算所有输入和输出位置的隐藏表示。在这些模型中，关联两个任意输入或输出位置的信号所需的操作数量随着位置之间的距离增长，对于 ConvS2S 是线性增长，对于 ByteNet 是指数增长。这使得学习远距离位置之间的依赖关系变得更加困难 [ 12]。在 Transformer 中，这被简化为一个常数数量的操作，尽管由于平均注意力加权位置而降低了有效分辨率，我们通过第 3.2 节描述的多头注意力（Multi-Head Attention）来对抗这一效应。

Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 27, 28, 22].
自注意力（Self-attention），有时也称为内部注意力（intra-attention），是一种注意力机制，它通过关联单个序列的不同位置来计算该序列的表示。自注意力已成功应用于多种任务，包括阅读理解、摘要生成、文本蕴含和学习任务无关的句子表示 [4, 27, 28, 22]。

End-to-end memory networks are based on a recurrent attention mechanism instead of sequence-aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [34].
端到端记忆网络基于循环注意力机制而非序列对齐的循环，并且已被证明在简单语言问答和语言建模任务上表现良好 [34]。

To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models such as [17, 18] and [9].
据我们所知，Transformer 是第一个完全依赖自注意力机制来计算其输入和输出的表示，而不使用序列对齐的 RNN 或卷积的转换模型。在接下来的章节中，我们将描述 Transformer，解释自注意力的动机，并讨论其相对于 [17, 18] 和 [9] 等模型的优势。

Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35]. Here, the encoder maps an input sequence of symbol representations $(x_{1},...,x_{n})$ to a sequence of continuous representations $\mathbf{z}=(z_{1},...,z_{n})$. Given $\mathbf{z}$, the decoder then generates an output sequence $(y_{1},...,y_{m})$ of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.
大多数竞争力的神经序列转换模型都采用编码器-解码器结构 [5, 2, 35]。在这里，编码器将输入的符号表示序列 $(x_{1},...,x_{n})$ 映射到一个连续表示序列 $\mathbf{z}=(z_{1},...,z_{n})$ 。给定 $\mathbf{z}$ ，解码器随后生成一个输出符号序列 $(y_{1},...,y_{m})$ ，每次生成一个元素。在每个步骤中，模型是自回归的 [10]，在生成下一个符号时会将之前生成的符号作为额外的输入。

The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively.
Transformer 遵循这种整体架构，使用堆叠的自注意力机制和逐点、全连接层，分别用于编码器和解码器，如图 1 的左半部分和右半部分所示。

In this section we compare various aspects of self-attention layers to the recurrent and convolutional layers commonly used for mapping one variable-length sequence of symbol representations $(x_{1},...,x_{n})$ to another sequence of equal length $(z_{1},...,z_{n})$, with $x_{i},z_{i}\in\mathbb{R}^{d}$, such as a hidden layer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we consider three desiderata.
在本节中，我们将自注意力层（self-attention layers）的各个方面与常用于将一个可变长度的符号表示序列 $(x_{1},...,x_{n})$ 映射到另一个等长序列 $(z_{1},...,z_{n})$ 的循环层（recurrent layers）和卷积层（convolutional layers）进行比较，其中 $x_{i},z_{i}\in\mathbb{R}^{d}$ ，例如在典型的序列转导编码器或解码器中的隐藏层。激发我们使用自注意力的原因，我们考虑了三个期望（desiderata）。

One is the total computational complexity per layer. Another is the amount of computation that can be parallelized, as measured by the minimum number of sequential operations required.
一个是每层的总计算复杂度。另一个是可以并行化的计算量，用所需的最小顺序操作次数来衡量。

The third is the path length between long-range dependencies in the network. Learning long-range dependencies is a key challenge in many sequence transduction tasks. One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare the maximum path length between any two input and output positions in networks composed of the different layer types.
第三是网络中长距离依赖之间的路径长度。学习长距离依赖是许多序列转换任务中的一个关键挑战。影响学习这种依赖能力的一个关键因素是前向和后向信号在网络中必须穿越的路径长度。这些路径在输入和输出序列的任何组合位置之间的长度越短，学习长距离依赖就越容易 [12]。因此，我们还比较了由不同层类型组成的网络中任意两个输入和输出位置之间的最长路径长度。 (EXPLANATION: - long-range dependencies (长距离依赖): 指的是在序列数据中，两个元素之间的距离较远，但它们之间仍然存在某种关系或依赖性。 - sequence transduction tasks (序列转换任务): 指将一种序列（如文本、语音）转换为另一种序列的任务，例如机器翻译、语音识别等。 - forward and backward signals (前向和后向信号): 在神经网络中，前向信号是指从输入到输出的信号传递过程，后向信号是指在训练过程中通过反向传播算法进行的误差传递。 - path length (路径长度): 在网络中，指的是从一个节点到另一个节点所需经过的边的数量或距离。)

As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires $O(n)$ sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length $n$ is smaller than the representation dimensionality $d$, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [38] and byte-pair [31] representations. To improve computational performance for tasks involving very long sequences, self-attention could be restricted to considering only a neighborhood of size $r$ in the input sequence centered around the respective output position. This would increase the maximum path length to $O(n/r)$. We plan to investigate this approach further in future work.
如表 1 所示，自注意力层以恒定数量的顺序执行操作连接所有位置，而循环层则需要 $O(n)$ 个顺序操作。在计算复杂度方面，当序列长度 $n$ 小于表示维度 $d$ 时，自注意力层比循环层更快，这在机器翻译中最先进模型使用的句子表示中通常是这种情况，例如词片[ 38]和字节对[ 31]表示。为了提高涉及非常长序列的任务的计算性能，可以限制自注意力仅考虑输入序列中以相应输出位置为中心的 $r$ 大小的邻域。这将使最大路径长度增加到 $O(n/r)$ 。我们计划在未来的工作中进一步研究这种方法。

A single convolutional layer with kernel width $k<n$ does not connect all pairs of input and output positions. Doing so requires a stack of $O(n/k)$ convolutional layers in the case of contiguous kernels, or $O(log_{k}(n))$ in the case of dilated convolutions [18], increasing the length of the longest paths between any two positions in the network. Convolutional layers are generally more expensive than recurrent layers, by a factor of $k$. Separable convolutions [6], however, decrease the complexity considerably, to $O(k\cdot n\cdot d+n\cdot d^{2})$. Even with $k=n$, however, the complexity of a separable convolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer, the approach we take in our model.
一个卷积层，核宽度为 $k<n$ ，并不会连接所有输入和输出位置的对。要做到这一点，在连续核的情况下，需要堆叠 $O(n/k)$ 个卷积层，或者在扩张卷积的情况下，需要 $O(log_{k}(n))$ 个卷积层[18]，这增加了网络中任意两个位置之间最长路径的长度。卷积层通常比循环层更昂贵，成本高出 $k$ 倍。然而，可分离卷积[6]大大降低了复杂性，降至 $O(k\cdot n\cdot d+n\cdot d^{2})$ 。即便如此，可分离卷积的复杂度也等于自注意力层和逐点前馈层的组合，这是我们在模型中采用的方法。

As side benefit, self-attention could yield more interpretable models. We inspect attention distributions from our models and present and discuss examples in the appendix. Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences.
作为副作用，自注意力机制可以产生更易于解释的模型。我们检查了模型中的注意力分布，并在附录中展示和讨论了示例。不仅单个注意头明显学会了执行不同的任务，许多注意头还表现出与句子的句法和语义结构相关的行为。

This section describes the training regime for our models.
本节描述了我们模型的训练方案。

In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.
在本工作中，我们提出了 Transformer，这是第一个完全基于注意力机制的序列转换模型，它用多头自注意力机制取代了编码器-解码器架构中最常用的循环层。

For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles.
在翻译任务中，Transformer 的训练速度比基于循环或卷积层的架构快得多。在 WMT 2014 英德翻译任务和 WMT 2014 英法翻译任务上，我们达到了新的最先进水平。在前者任务中，我们的最佳模型甚至超过了之前报道的所有集成模型。 (WMT 2014: Workshop on Machine Translation 2014)

We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video. Making generation less sequential is another research goals of ours.
我们对基于注意力机制的模型的未来感到兴奋，并计划将其应用于其他任务。我们计划将 Transformer 扩展到处理除文本之外的其他输入和输出模式的问题，并研究局部、受限的注意力机制，以高效处理大规模输入和输出，如图像、音频和视频。使生成过程更少依赖序列是我们的另一个研究目标。

The code we used to train and evaluate our models is available at https://github.com/tensorflow/tensor2tensor.
我们用来训练和评估模型的代码可以在以下链接找到：https://github.com/tensorflow/tensor2tensor。