MaskGIT: Masked Generative Image Transformer
MaskGIT：掩码生成式图像变换器

Deep generative models [29, 45, 17, 53, 41, 12, 46, 34] have achieved lots of successes in image synthesis tasks. GAN based methods demonstrate amazing capability in yielding high-fidelity samples [17, 4, 27, 53, 44]. In contrast, likelihood-based methods, such as Variational Autoencoders (VAEs) [29, 45], Diffusion Models [41, 12, 24] and Autoregressive Models [46, 34], offer distribution coverage and hence can generate more diverse samples [41, 45, 46].
深度生成模型[29,45,17,53,41,12,46,34]在图像合成任务中取得了诸多成功。基于 GAN 的方法在生成高保真样本方面展现出惊人能力[17,4,27,53,44]。相比之下，基于似然的方法——如变分自编码器(VAEs)[29,45]、扩散模型[41,12,24]和自回归模型[46,34]——能提供分布覆盖，因此可生成更多样化的样本[41,45,46]。

However, maximizing likelihood directly in pixel space can be challenging. So instead, VQVAE [47, 37] proposes to generate images in latent space in two stages. In the first stage, which is known as tokenization, it tries to compress images into discrete latent space, and primarily consists of three components:
然而直接在像素空间最大化似然具有挑战性。为此 VQVAE[47,37]提出分两阶段在潜在空间生成图像：第一阶段称为标记化(tokenization)，通过三个主要组件将图像压缩至离散潜在空间：

• •

  an encoder $E$ that learns to tokenize images $x\in\mathbb{R}^{H\times W\times 3}$ into latent embedding $E(x)$,
  一个编码器 $E$ 学习将图像 $x\in\mathbb{R}^{H\times W\times 3}$ 转换为潜在嵌入 $E(x)$ ，

• •

  a codebook $\mathbf{e}_{k}\in\mathbb{R}^{D},k\in 1,2,\cdots,K$ which serves for a nearest neighbor look up used to quantize the embedding into visual tokens, and
  一个用于最近邻查找的码本 $\mathbf{e}_{k}\in\mathbb{R}^{D},k\in 1,2,\cdots,K$ ，将嵌入向量量化为视觉标记

• •

  a decoder $G$ which predicts the reconstructed image $\hat{x}$ from the visual tokens $\mathbf{e}$.
  一个解码器 $G$ ，用于根据视觉标记 $\mathbf{e}$ 预测重建图像 $\hat{x}$

The transformer architecture [48], was first proposed in NLP, and has recently extended its reach to computer vision [13, 6]. Transformer consists of multiple self-attention layers, allowing interactions between all pairs of elements in the sequence to be captured. In particular, BERT [11] introduces the masked language modeling (MLM) task for language representation learning. The bi-directional self-attention used in BERT [11] allows the masked tokens in MLM to be predicted utilizing context from both directions. In vision, the masked modeling in BERT [11] has been extended to image representation learning [21, 2] with images quantized to discrete tokens. However, few works have successfully applied the same masked modeling to image generation [56] because of the difficulty in performing autoregressive decoding using bi-directional attentions. To our knowledge, this paper provides the first evidence demonstrating the efficacy of masked modeling for image generation on the common ImageNet benchmark. Our work is inspired by bi-directional machine translation [16, 20, 19] in NLP, and our novelty lies in the proposed new masking strategy and decoding algorithm which, as substantiated by our experiments, are essential for image generation.
Transformer 架构[48]最初在自然语言处理领域提出，近年来已扩展至计算机视觉领域[13,6]。该架构由多个自注意力层组成，能捕捉序列中所有元素对之间的交互关系。特别是 BERT[11]提出的掩码语言建模(MLM)任务，通过双向自注意力机制使掩码标记能利用上下文双向信息进行预测。在视觉领域，研究者将 BERT[11]的掩码建模思想拓展至图像表示学习[21,2]，将图像量化为离散标记。然而由于双向注意力机制在自回归解码上的困难，鲜有研究能成功将相同掩码建模应用于图像生成任务[56]。据我们所知，本文首次在通用 ImageNet 基准上验证了掩码建模对图像生成的有效性。 我们的工作受到自然语言处理中双向机器翻译[16,20,19]的启发，创新点在于提出的新型掩码策略和解码算法——实验证明这些改进对图像生成至关重要。

Let ${\mathbf{Y}}=[y_{i}]_{i=1}^{N}$ denote the latent tokens obtained by inputting the image to the VQ-encoder, where $N$ is the length of the reshaped token matrix, and ${\mathbf{M}}=[m_{i}]_{i=1}^{N}$ the corresponding binary mask. During training, we sample a subset of tokens and replace them with a special [MASK] token. The token $y_{i}$ is replaced with [MASK] if $m_{i}=1$, otherwise, when $m_{i}=0$, $y_{i}$ will be left intact.
令 ${\mathbf{Y}}=[y_{i}]_{i=1}^{N}$ 表示通过将图像输入 VQ 编码器获得的潜在标记，其中 $N$ 是重塑后标记矩阵的长度， ${\mathbf{M}}=[m_{i}]_{i=1}^{N}$ 为对应的二元掩码。在训练过程中，我们采样一个标记子集并用特殊的[MASK]标记替换它们。当 $m_{i}=1$ 时，标记 $y_{i}$ 会被替换为[MASK]；反之当 $m_{i}=0$ 时， $y_{i}$ 将保持原状。

The sampling procedure is parameterized by a mask scheduling function $\gamma(r)\in(0,1]$, and executes as follows: we first sample a ratio from $0$ to $1$, then uniformly select $\lceil\gamma(r)\cdot N\rceil$ tokens in ${\mathbf{Y}}$ to place masks, where $N$ is the length. The mask scheduling significantly affects the quality of image generation and will be discussed in 3.3.
采样过程由掩码调度函数 $\gamma(r)\in(0,1]$ 参数化，并按如下方式执行：首先从 $0$ 到 $1$ 采样一个比例值，然后在 ${\mathbf{Y}}$ 中均匀选择 $\lceil\gamma(r)\cdot N\rceil$ 个标记放置掩码，其中 $N$ 表示长度。掩码调度策略会显著影响图像生成质量，我们将在 3.3 节详细讨论。

Denote $Y_{{\overline{\mathbf{M}}}}$ the result after applying mask ${\mathbf{M}}$ to ${\mathbf{Y}}$. The training objective is to minimize the negative log-likelihood of the masked tokens:
设 $Y_{{\overline{\mathbf{M}}}}$ 为对 ${\mathbf{Y}}$ 应用掩码 ${\mathbf{M}}$ 后的结果。训练目标是最小化被掩码标记的负对数似然：

In autoregressive decoding, tokens are generated sequentially based on previously generated output. This process is not parallelizable and thus very slow for image because the image token length, e.g. 256 or 1024, is typically much larger than that of language. We introduce a novel decoding method where all tokens in the image are generated simultaneously in parallel. This is feasible due to the bi-directional self-attention of MTVM.
在自回归解码过程中，标记是基于先前生成的输出顺序生成的。这一过程无法并行化，因此对于图像生成而言速度极慢，因为图像标记长度（例如 256 或 1024）通常远大于语言标记长度。我们提出了一种新颖的解码方法，其中图像中的所有标记都能并行同步生成。这种并行化可行性得益于 MTVM 架构的双向自注意力机制。

In theory, our model is able to infer all tokens and generate the entire image in a single pass. We find this challenging due to inconsistency with the training task. Below, the proposed iterative decoding is introduced. To generate an image at inference time, we start from a blank canvas with all the tokens masked out, i.e. $Y_{\mathbf{M}}^{(0)}$. For iteration $t$, our algorithm runs as follows:
理论上，我们的模型能够一次性推断所有标记并生成完整图像。但由于与训练任务存在不一致性，我们发现这一目标具有挑战性。下文将介绍所提出的迭代解码方法。在推理阶段生成图像时，我们从所有标记被掩蔽的空白画布（即 $Y_{\mathbf{M}}^{(0)}$ ）开始。对于迭代 $t$ ，算法运行流程如下：

1. 1.

   Predict. Given the masked tokens $Y_{\mathbf{M}}^{(t)}$ at the current iteration, our model predicts the probabilities, denoted as $p^{(t)}\in\mathbb{R}^{N\times K}$, for all the masked locations in parallel.
   预测。给定当前迭代中的掩码标记 $Y_{\mathbf{M}}^{(t)}$ ，我们的模型并行预测所有掩码位置的概率，记为 $p^{(t)}\in\mathbb{R}^{N\times K}$ 。
   

2. 2.

   Sample. At each masked location $i$, we sample a token $y_{i}^{(t)}$ based on its prediction probabilities $p_{i}^{(t)}\in\mathbb{R}^{K}$ over all possible tokens in the codebook. After a token $y_{i}^{(t)}$ is sampled, its corresponding prediction score is used as a “confidence” score indicating the model’s belief of this prediction. For the unmasked position in $Y_{\mathbf{M}}^{(t)}$, we simply set its confidence score to $1.0$.
   采样。在每个掩码位置 $i$ ，我们根据其在码本中所有可能标记上的预测概率 $p_{i}^{(t)}\in\mathbb{R}^{K}$ 来采样一个标记 $y_{i}^{(t)}$ 。当标记 $y_{i}^{(t)}$ 被采样后，其对应的预测分数将作为"置信度"分数，表示模型对该预测的确信程度。对于 $Y_{\mathbf{M}}^{(t)}$ 中的非掩码位置，我们直接将其置信度分数设为 $1.0$ 。
   

3. 3.

   Mask Schedule. We compute the number of tokens to mask according to the mask scheduling function $\gamma$ by $n=\lceil\gamma(\frac{t}{T})N\rceil$, where $N$ is the input length and $T$ is the total number of iterations.
   掩码调度。我们根据掩码调度函数 $\gamma$ ，通过 $n=\lceil\gamma(\frac{t}{T})N\rceil$ 计算需要掩码的标记数量，其中 $N$ 是输入长度， $T$ 是总迭代次数。

4. 4.

   Mask. We obtain $Y_{\mathbf{M}}^{(t+1)}$ by masking $n$ tokens in $Y_{\mathbf{M}}^{(t)}$. The mask ${\mathbf{M}}^{(t+1)}$ for iteration $t+1$ is calculated from:

   $$m_{i}^{(t+1)}=\begin{cases}1,&\text{if $c_{i}<{\text{sorted}}_{j}(c_{j})[n]$.}\\ 0,&\text{otherwise.}\vspace{-2mm}\end{cases},$$

   where $c_{i}$ is the confidence score for the $i$-th token.

   
   掩码处理。我们通过在 $Y_{\mathbf{M}}^{(t)}$ 中掩码 $n$ 个标记来获得 $Y_{\mathbf{M}}^{(t+1)}$ 。第 $t+1$ 次迭代的掩码 ${\mathbf{M}}^{(t+1)}$ 由以下公式计算得出：其中 $c_{i}$ 表示第 $i$ 个标记的置信度分数。

The decoding algorithm synthesizes an image in $T$ steps. At each iteration, the model predicts all tokens simultaneously but only keeps the most confident ones. The remaining tokens are masked out and re-predicted in the next iteration. The mask ratio is made decreasing until all tokens are generated within $T$ iterations. In practice, the masking tokens are randomly sampled with temperature annealing to encourage more diversity, and we will discuss its effect in 3. Figure 2 illustrates an example of our decoding process. It generates an image in $T=8$ iterations, where the unmasked tokens at each iteration are highlighted in the grid, e.g. when $t=1$ we only keep 1 token and mask out the rest.
解码算法通过 $T$ 个步骤合成图像。在每次迭代中，模型同时预测所有标记，但仅保留置信度最高的部分。其余标记会被遮蔽并在下一次迭代中重新预测。掩码比例会逐步降低，直到所有标记在 $T$ 次迭代内生成完毕。实际操作中，我们会采用温度退火方法随机采样遮蔽标记以增强多样性，其效果将在第 3 节讨论。图 2 展示了我们解码过程的示例：该过程通过 $T=8$ 次迭代生成图像，每次迭代中未被遮蔽的标记会在网格中高亮显示（例如当 $t=1$ 时仅保留 1 个标记并遮蔽其余部分）。

We find that the quality of image generation is significantly affected by the masking design. We model the masking procedure by a mask scheduling function $\gamma(\cdot)$ that computes the mask ratio for the given latent tokens. As discussed, the function $\gamma$ is used in both training and inference. During inference time, it takes the input of $0/T,1/T,\cdots,(T-1)/T$ indicating the progress in decoding. In training, we randomly sample a ratio $r$ in $[0,1)$ to simulate the various decoding scenarios.
我们发现图像生成质量显著受掩码设计影响。我们通过掩码调度函数 $\gamma(\cdot)$ 对掩码过程进行建模，该函数计算给定潜在标记的掩码比例。如前所述，函数 $\gamma$ 在训练和推理阶段均被使用。在推理过程中，它以 $0/T,1/T,\cdots,(T-1)/T$ 作为输入来指示解码进度。训练时，我们在 $[0,1)$ 范围内随机采样比例 $r$ 以模拟各种解码场景。

BERT uses a fixed mask ratio of 15% [11], i.e., it always masks 15% of the tokens, which is unsuitable for our task since our decoder needs to generate images from scratch. New masking scheduling is thus needed. Before discussing specific schemes, we first examine the property of the mask scheduling function. First, $\gamma(r)$ needs to be a continuous function bounded between $0$ and $1$ for $r\in[0,1]$. Second, $\gamma(r)$ should be (monotonically) decreasing with respect to $r$, and it holds that $\gamma(0)\rightarrow 1$ and $\gamma(1)\rightarrow 0$. The second property ensures the convergence of our decoding algorithm.
BERT 采用固定的 15%掩码比例[11]，即始终遮蔽 15%的标记，这不适用于我们的任务，因为我们的解码器需要从零开始生成图像。因此需要新的掩码调度方案。在讨论具体方案前，我们首先研究掩码调度函数的特性。首先， $\gamma(r)$ 需要是定义在 $r\in[0,1]$ 区间内、取值介于 $0$ 和 $1$ 之间的连续函数。其次， $\gamma(r)$ 应关于 $r$ （单调）递减，且满足 $\gamma(0)\rightarrow 1$ 和 $\gamma(1)\rightarrow 0$ 。第二个特性确保了解码算法的收敛性。

This paper considers common functions and makes simple transformations so that they satisfy the properties. Figure 8 visualizes these functions which are divided into three groups:
本文对常见函数进行简单变换以满足特性要求。图 8 将这些函数可视化展示，分为三类：

• •

  Linear function is a straightforward solution, which masks an equal amount of tokens each time.
  线性函数是最直接的解决方案，每次遮蔽等量的标记。

• •

  Concave function captures the intuition that image generation follows a less-to-more information flow. In the beginning, most tokens are masked so the model only needs to make a few correct predictions for which the model feel confident. Towards the end, the mask ratio sharply drops, forcing the model to make a lot more correct predictions. The effective information is increasing in this process. The concave family includes cosine, square, cubic, and exponential.
  凹函数捕捉了图像生成遵循"从少到多"信息流的直觉。初期大多数标记被遮蔽，模型只需做出少量有把握的正确预测；接近尾声时遮蔽率急剧下降，迫使模型做出更多正确预测。这一过程中有效信息持续增加。凹函数族包括余弦、平方、立方和指数函数。

• •

  Convex function, conversely, implements a more-to-less process. The model needs to finalize a vast majority of tokens within the first couple of iterations. This family includes square root and logarithmic.
  凸函数则相反，实现"从多到少"的过程。模型需要在最初几次迭代中确定绝大多数标记。该函数族包括平方根和对数函数。

We empirically compare the above mask scheduling functions in 3 and find the cosine function works the best in all of our experiments.
我们通过实验比较了上述三种掩码调度函数，发现余弦函数在所有实验中表现最佳。

For each dataset, we only train a single autoencoder, decoder, and codebook with 1024 tokens on cropped 256x256 images for all the experiments. The image is always compressed by a fixed factor of 16, i.e. from $H\times W$ to a grid of tokens in the size of $h\times w$, where $h$=$H/16$ and $w$=$W/16$. We find that this autoencoder, together with the codebook, can be reused to synthesize 512$\times$512 images.
针对每个数据集，我们仅训练一个自动编码器、解码器和包含 1024 个标记的码本，所有实验均在裁剪后的 256x256 图像上进行。图像始终按固定比例 16 进行压缩，即从 $H\times W$ 尺寸压缩至 $h\times w$ 大小的标记网格，其中 $h$ = $H/16$ 且 $w$ = $W/16$ 。我们发现该自动编码器与码本可复用于合成 512 $\times$ 512 尺寸的图像。

All models in this work have the same configuration: 24 layers, 8 attention heads, 768 embedding dimensions and 3072 hidden dimensions. Our models use learnable positional embedding[48], LayerNorm[1], and truncated normal initialization (stddev=$0.02$). We employ the following training hyperparameters: label smoothing=$0.1$, dropout rate=$0.1$, Adam optimizer [28] with $\beta_{1}$=$0.9$ and $\beta_{2}$=$0.96$. We use RandomResizeAndCrop for data augmentation. All models are trained on 4x4 TPU devices with a batch size of 256. ImageNet models are trained for 300 epochs while the Places2 model is trained for 200 epochs.
本研究中所有模型采用相同配置：24 层网络结构、8 个注意力头、768 维嵌入空间和 3072 维隐藏层。模型使用可学习的位置嵌入[48]、层归一化[1]以及截断正态初始化（标准差= $0.02$ ）。训练超参数设置为：标签平滑= $0.1$ 、丢弃率= $0.1$ 、采用 Adam 优化器[28]（ $\beta_{1}$ = $0.9$ ， $\beta_{2}$ = $0.96$ ）。数据增强采用随机缩放裁剪策略。所有模型均在 4x4 TPU 设备上以 256 批次规模训练，其中 ImageNet 模型训练 300 轮次，Places2 模型训练 200 轮次。

We evaluate the performance of our model on class-conditional image synthesis on ImageNet 256$\times$256 and 512$\times$512. Our main results are summarized in Table 1.
我们在 ImageNet 256×256 和 512×512 分辨率下评估了模型在类别条件图像合成中的性能，主要结果汇总于表 1。

We also train a VQGAN baseline with the same tokenizer and hyperparameters as MaskGIT’s in order to further highlight the difference between bi-directional and uni-directional transformers, and find that on both resolutions, MaskGIT still outperforms our implemented baseline by a significant margin.
为突显双向与单向 Transformer 的差异，我们使用与 MaskGIT 相同的分词器和超参数训练了 VQGAN 基线模型。实验表明，在两种分辨率下，MaskGIT 仍以显著优势超越我们实现的基线模型。

Furthermore, MaskGIT improves BigGAN’s FIDs on both resolutions, achieving a new state-of-the-art on 512$\times$512 with an FID of $7.32$.
此外，MaskGIT 在两种分辨率下均改进了 BigGAN 的 FID 指标，其中在 512×512 分辨率上以 $7.32$ 的 FID 创造了新的最优记录。

To further substantiate the speed difference between MaskGIT and autoregressive models, we perform a runtime comparison between MaskGIT and VQGAN’s decoding processes. As illustrated in Figure 4, MaskGIT significantly accelerates VQGAN by $30$-$64$x, with a speedup that gets more pronounced as the image resolution (and thus the input token length) grows.
为进一步验证 MaskGIT 与自回归模型的速度差异，我们对 MaskGIT 与 VQGAN 的解码过程进行了运行时对比。如图 4 所示，MaskGIT 将 VQGAN 加速了 $30$ 至 $64$ 倍，且随着图像分辨率（即输入标记长度）的增加，这种加速效果愈发显著。

CAS involves first training a ResNet-50 classifier[22] solely on the samples generated by the candidate model, and then measuring the classifier’s classification accuracy on the ImageNet validation set. The last two columns in Table 1 present the CAS results, where the scores of the classifier trained on real ImageNet training data are included for reference (76.6% and 93.1% for the top-1 and top-5 accuracy). For image resolution 256$\times$256, we follow the common practice of using data augmentation RandAugment[9], and report the scores trained without augmentation in the appendix  B. We find that MaskGIT significantly outperforms prior work VQVAE-2 and VQGAN, establishing a new state-of-the-art of CAS on the ImageNet benchmark on both resolutions.
CAS（分类器精度评分）首先需要训练一个仅基于候选模型生成样本的 ResNet-50 分类器[22]，然后测量该分类器在 ImageNet 验证集上的分类准确率。表 1 最后两列展示了 CAS 结果，其中包含基于真实 ImageNet 训练数据训练的分类器准确率作为参考（top-1 准确率 76.6%，top-5 准确率 93.1%）。对于 256×256 分辨率图像，我们遵循常规做法使用 RandAugment 数据增强[9]，未使用数据增强的训练结果详见附录 B。研究发现，MaskGIT 显著优于先前工作 VQVAE-2 和 VQGAN，在两种分辨率下均创下了 ImageNet 基准测试中 CAS 指标的新纪录。

The Precision/Recall results in Table 1 show that MaskGIT achieves better coverage (Recall) compared to BigGAN, and better sample quality (Precision) compared to likelihood-based models such as VQVAE-2 and diffusion models. Compared to our baseline VQGAN, we improve the diversity as measured by recall while slightly boosting its precision.
表 1 中的精确率/召回率结果显示，与 BigGAN 相比，MaskGIT 实现了更好的覆盖率（召回率）；与 VQVAE-2 和扩散模型等基于似然的模型相比，则具有更优的样本质量（精确率）。相较于基线模型 VQGAN，我们在保持精确率小幅提升的同时，通过召回率指标衡量显著提高了生成多样性。

In contrast to BigGAN’s samples, MaskGIT’s samples are more diverse with more varied lighting, poses, scales and context as shown in Figure 5. More comparisons are available in the appendix  B.
与 BigGAN 的样本相比，如图 5 所示，MaskGIT 生成的样本在光照、姿态、比例和场景方面展现出更丰富的多样性。更多对比结果详见附录 B。

In this subsection, we present direct applications of MaskGIT on three image editing tasks: class-conditional image editing, image inpainting, and outpainting. All three tasks can be almost trivially translated to ones that MaskGIT can handle if we consider the task as just a constraint on the initial binary mask ${\mathbf{M}}$ MaskGIT uses in its iterative decoding, as discussed in  3.2. We show that without modifications to the architecture or any task-specific training, MaskGIT is capable of generating very compelling results on all three applications. Furthermore, MaskGIT obtains comparable performance to dedicated models on both inpainting and outpainiting, even though it is not designed specifically for either task.
在本小节中，我们将展示 MaskGIT 在三种图像编辑任务上的直接应用：类别条件图像编辑、图像修复和外绘。正如 3.2 节所述，若将这些任务视为对初始二值掩码 ${\mathbf{M}}$ 的约束条件（MaskGIT 在迭代解码过程中使用），那么这三个任务几乎都能直接转化为 MaskGIT 可处理的形式。研究表明，无需修改架构或进行任何任务特定训练，MaskGIT 就能在所有三项应用中生成极具吸引力的结果。更值得注意的是，尽管 MaskGIT 并非专为修复或外绘任务设计，但其在这两项任务上的表现与专用模型相比毫不逊色。

For MaskGIT, however, it is a trivial task if we consider the bounding box region as the input of initial mask to the iterative decoding algorithm. Figure 6 shows a few example results. More can be found in the appendix  C.
然而对于 MaskGIT 而言，若将边界框区域作为初始掩码输入迭代解码算法，这不过是轻而易举的任务。图 6 展示了若干示例结果，更多案例可参阅附录 C。

In these examples, we observe that MaskGIT can reasonably replace the selected object while preserving, or to some extend even completing, the context in the background. Furthermore, we find that MaskGIT seems to be capable of synthesizing unnatural yet plausible combinations unseen in the ImageNet training set, e.g. a flying cat, cat in a soup bowl, and cat in a flower. This suggests that MaskGIT has incidentally learned useful representations for composition, which may be further exploited in related tasks in future works.
在这些示例中，我们观察到 MaskGIT 能够合理替换选定对象，同时保持（甚至在一定程度上补全）背景内容。更值得注意的是，MaskGIT 似乎能合成 ImageNet 训练集中未曾出现、看似反常却合理的组合——例如飞行中的猫、汤碗里的猫，以及花朵中的猫。这表明 MaskGIT 意外习得了有效的组合表征能力，未来或可在相关任务中进一步挖掘其潜力。

We extend MaskGIT to this problem by tokenizing the masked image and interpreting the inpainting mask as the initial mask in our iterative decoding. We then composite the output image by linearly blending it with the input based on the masking boundary following [8]. To match the training of our baselines, we train MaskGIT on the 512$\times$512 center-cropped images from the Places2[58] dataset. All hyperparameters are kept the same as the MaskGIT model trained on ImageNet.
我们通过将掩码图像进行标记化处理，并将修复掩码解释为迭代解码中的初始掩码，从而将 MaskGIT 扩展应用于此问题。随后按照文献[8]的方法，基于掩码边界对输出图像与输入图像进行线性混合合成。为与基线模型的训练保持一致，我们在 Places2[58]数据集的 512×512 中心裁剪图像上训练 MaskGIT 模型，所有超参数均保持与 ImageNet 上训练的 MaskGIT 模型相同。

We compare MaskGIT against common GAN-based baselines, including DeepFillv2[52] and HiFill[50], on inpainting with a central 50% $\times$ 50% mask, which are evaluated on the Places2 validation set. Table 2 summarizes the quantitative comparisons. MaskGIT beats both DeepFill and HiFill in FID and IS by a significant margin, while achieving scores close to the state-of-the-art inpainting approach CoModGAN [57]. We show more qualitative comparisons with CoModGAN in the appendix  E.
我们将 MaskGIT 与基于 GAN 的常见基线方法（包括 DeepFillv2[52]和 HiFill[50]）在中心区域 50%×50%掩码的图像修复任务上进行比较，评估基于 Places2 验证集进行。表 2 总结了定量对比结果。MaskGIT 在 FID 和 IS 指标上均以显著优势超越 DeepFill 和 HiFill，同时达到与当前最先进的图像修复方法 CoModGAN[57]相近的分数。更多与 CoModGAN 的定性对比结果详见附录 E。

We compare against common GAN-based baselines, including Boundless [43], In&Out [8], InfinityGAN[31], and CoModGAN[57] on extrapolating rightward with a 50% ratio. We evaluate on the image set generously provided by the authors of InfinityGAN[31] and In&Out[8].
我们与常见的基于 GAN 的基线方法进行了比较，包括 Boundless[43]、In&Out[8]、InfinityGAN[31]和 CoModGAN[57]，测试其在 50%比例下向右外推的性能。评估所用图像集由 InfinityGAN[31]和 In&Out[8]的作者慷慨提供。

Table 2 summarizes the quantitative comparisons. MaskGIT beats all baselines and achieves state-of-the-art FID and IS. As the examples in Figure 7 illustrate, MaskGIT is also capable of synthesizing diverse results given the same input with different seeds. We observe that MaskGIT completes objects and global structures particularly well, and hypothesize that this is thanks to the model learning useful representations with the global attentions in the transformer.
表 2 总结了量化对比结果。MaskGIT 在所有基线模型中表现最优，取得了当前最佳的 FID 和 IS 分数。如图 7 所示案例表明，在相同输入配合不同随机种子的情况下，MaskGIT 仍能生成多样化结果。我们注意到该模型在物体补全与全局结构重建方面表现尤为突出，推测这得益于 Transformer 架构中全局注意力机制所学习到的有效表征。

We conduct ablation experiments using the default setting on ImageNet 256$\times$256.
我们在 ImageNet 256×256 数据集上使用默认设置进行了消融实验。

We observe that concave functions generally obtain better FID and IS than linear, followed by the convex functions. While cosine and square perform similarly relative to other functions, cosine slightly edges out square in all scores, making cosine the default in our model.
我们观察到，凹函数通常能获得比线性函数更好的 FID 和 IS 分数，其次是凸函数。虽然余弦函数和平方函数相对于其他函数表现相似，但余弦函数在所有指标上都略微优于平方函数，因此我们选择余弦函数作为模型的默认设置。

We hypothesize that concave functions perform favorably because they 1) challenge training with more difficult cases (i.e. encouraging larger mask ratios), and 2) appropriately prioritize the less-to-more prediction throughout the decoding. That said, over-prioritization seems to be costly as well, as shown by the cubic function being worse than square, and exponential being much worse than all other concave functions.
我们推测凹函数表现优异的原因在于：1）通过更具挑战性的案例（即鼓励使用更大的掩码比例）来强化训练；2）在解码过程中合理分配从少到多的预测优先级。但过度优先化似乎也会带来负面影响，立方函数表现逊于平方函数，而指数函数更是远差于其他凹函数的结果印证了这一点。