Image Matching Filtering and Refinement
by Planes and Beyond
图像匹配滤波与平面及超越平面的优化

Keypoint detection is aimed at finding salient local image regions whose feature descriptor vectors are at the same time-invariant to image transformations, i.e. repeatable, and distinguishable from other non-corresponding descriptors, i.e. discriminant. Clearly, discriminability decreases as long as invariance increases and vice-versa. Keypoints are usually recognized as corner-like and blob-like ones, even if there is no clear separation within them outside ad-hoc ideal images. Corners are predominantly extracted by the Harris detector [62], while blobs by SIFT [19]. Several other handcrafted detectors exist besides the Harris and the SIFT one [63, 64, 65], but excluding deep image matching nowadays the former and their extensions are the most commonly adopted. Harris and SIFT keypoint extractors rely respectively on the covariance matrix of the first-order derivatives and the Hessian matrix of the second-order derivatives. Blobs and corners can be considered each other analogues for different derivative orders, as in the end, a keypoint in both cases correspond to a local maxima response to a filter which is a function of the determinant of the respective kind of matrix. Blob keypoint centers tend to be pushed away from edges, which characterize the regions, more than corners due to the usage of high-order derivatives. More recently, the joint extraction and refinement of robust keypoints has been investigated by the Gaussian Mixture Models for Interpretable Keypoint Refinement and Scoring (GMM-IKRS) [66]. The idea is to extract image keypoints after several homography warps and cluster them after back-projection into the input images so as to select the best clusters in terms of robustness and deviation, as another perspective of the Affine SIFT (ASIFT) [67] strategy.
关键点检测旨在寻找显著的局部图像区域，其特征描述符向量同时对图像变换具有时间不变性，即重复性，并且能与其他非对应描述符区分开来，即具有判别性。显然，判别性随不变性的增加而降低，反之亦然。关键点通常被识别为角点状和斑点状，尽管在理想图像之外，它们之间没有明确的区分。角点主要通过 Harris 检测器[62]提取，而斑点则通过 SIFT[19]提取。除了 Harris 和 SIFT 之外，还存在其他几种手工设计的检测器[63, 64, 65]，但如今在深度图像匹配之外，前者及其扩展是最常用的。Harris 和 SIFT 关键点提取器分别依赖于一阶导数的协方差矩阵和二阶导数的 Hessian 矩阵。斑点和角点可以被视为不同导数阶数的对应物，因为在最终情况下，两种情况下的关键点都对应于对某种矩阵行列式函数的滤波器的局部最大响应。 Blob 关键点中心往往被推离边缘，这些边缘特征区域比角点更明显，这是由于使用了高阶导数。最近，通过高斯混合模型进行可解释关键点精化和评分（GMM-IKRS）[66]，研究了鲁棒关键点的联合提取与精化。其思路是在多次单应性变换后提取图像关键点，并将它们反投影回输入图像进行聚类，从而根据鲁棒性和偏差选择最佳聚类，这是仿射 SIFT（ASIFT）[67]策略的另一种视角。

Feature descriptors aim at embedding the keypoint characterization into numerical vectors. There has been a long research interest in this area, both in terms of handcrafted methods and non-deep machine learning ones, which contributed actively to the current development and the progress made by current SOTA deep image matching. Among non-deep keypoint descriptors, nowadays only the SIFT descriptor is practically still used and widespread due to its balance within computational efficiency, scaling, and matching robustness. SIFT descriptor is defined as the gradient orientation histogram of the local neighborhood of the keypoint, also referred to as patch. Many high-level computer vision applications like COLMAP [4] in the case of SfM are based on SIFT. Besides SIFT keypoint detector & descriptor, the Oriented FAST and Rotated BRIEF (ORB) [68] is a pipeline surviving still today in the deep era worth to be mentioned. ORB is the core of the popular ORB-SLAM [6], due to its computational efficiency, even if less robust than SIFT. Nevertheless, it seems to be going to be replaced by SIFT with the increased computational capability of current hardware [69].
特征描述符旨在将关键点特征嵌入数值向量中。长期以来，这一领域吸引了大量研究兴趣，涵盖手工设计方法和非深度机器学习方法，它们积极推动了当前深度图像匹配技术的发展与进步。在非深度关键点描述符中，如今仅 SIFT 描述符因其计算效率、尺度适应性和匹配鲁棒性之间的平衡而实际应用广泛。SIFT 描述符定义为关键点局部邻域的梯度方向直方图，亦称为图像块。众多高级计算机视觉应用，如 SfM 中的 COLMAP[4]，均基于 SIFT。除 SIFT 关键点检测器与描述符外，Oriented FAST 与 Rotated BRIEF（ORB）[68]这一流程在深度学习时代仍值得提及，它作为 ORB-SLAM[6]的核心，因其计算效率而流行，尽管鲁棒性不及 SIFT。然而，随着当前硬件计算能力的提升，ORB 似乎正被 SIFT 所取代[69]。

Actually, the patch normalization takes place in between the detection and description of the keypoints, and it is critical for effective computation of the descriptor to be associated with the keypoint. Patch normalization is implicitly hidden inside both keypoint extraction and description and its task roughly consists of aligning and registering patches before descriptor computation [19]. The key idea is that complex image deformations can be locally approximated by more simple transformations as long as the considered neighborhood is small. Under these assumptions, SIFT normalizes the patch by a similarity transformation, where the scale is inferred by the detector and the dominant rotation is computed from the gradient orientation histogram. Affine covariant patch normalization [70, 67, 71] was indeed the natural evolution, which can improve the normalization in case of high degrees of perspective distortions. Notice that in the case of only small scene rotation, up-right patches computed neglecting orientation alignment can lead to better matches since a strong geometric constraint limiting the solution space is imposed [72]. Concerning instead patch photometric consistency, affine normalization, i.e. by mean and standard deviation, is usually the preferred choice being sufficiently robust in the general case [19].
实际上，补丁归一化发生在关键点检测与描述之间，对于有效计算与关键点相关联的描述符至关重要。补丁归一化隐含在关键点提取和描述过程中，其任务大致包括在描述符计算前对齐和注册补丁[19]。核心思想是，只要考虑的邻域足够小，复杂的图像形变可以在局部近似为更简单的变换。在这些假设下，SIFT 通过相似性变换对补丁进行归一化，其中尺度由检测器推断，主旋转则从梯度方向直方图中计算得出。仿射协变补丁归一化[70, 67, 71]确实是自然演进，能够在存在高度透视畸变的情况下改进归一化效果。值得注意的是，在场景旋转较小的情况下，忽略方向对齐计算的直立补丁可能会带来更好的匹配结果，因为强几何约束限制了解空间[72]。 关于补丁光度一致性，仿射归一化，即通过均值和标准差进行归一化，通常是首选方法，因为它在一般情况下足够稳健[19]。

The last step is the proper matching, which mainly pairs keypoints by considering the distance similarity on the associated descriptors. Nearest Neighbor (NN) and Nearest Neighbor Ratio (NNR) [19] are the common and simple approaches even if additional extensions have been developed [73, 32]. For many tasks descriptor similarity alone does not provide satisfactory results, which can be achieved instead by including a further filtering post-process by constraining matches geometrically. RANSAC provides an effective method to discard outliers under the more general epipolar geometry constraints of the scene or in the case of planar homographies. The standard RANSAC continues to be extended in several ways [21, 22]. Worth mentioning are the Degenerate SAmpling Consensus (DegenSAC) [24] to avoid degenerate configurations, and the MArGinalized SAmpling Consensus (MAGSAC) [23] for defining a robust inlier threshold which remains the most critical parameter.
最后一步是适当的匹配，主要通过考虑关联描述符上的距离相似性来配对关键点。最近邻（NN）和最近邻比率（NNR）[19]是常见且简单的匹配方法，尽管已开发出其他扩展方法[73, 32]。对于许多任务，仅依赖描述符相似性往往无法获得满意的结果，而通过几何约束进行进一步过滤的后处理步骤则能实现更好的匹配效果。RANSAC 提供了一种有效的方法，在场景的更普遍的极线几何约束下或平面单应性的情况下，剔除异常值。标准 RANSAC 在多个方面得到了扩展[21, 22]。值得一提的是，退化采样一致性（DegenSAC）[24]用于避免退化配置，而边缘化采样一致性（MAGSAC）[23]则用于定义一个鲁棒的内点阈值，这是最关键的参数。

RANSAC exploits global geometric constraints. A more general and relaxed geometric assumption than RANSAC is to consider only local neighborhood consistency across the images as done in Grid-based Motion Statistics (GMS) [26] using square neighborhood, but also circular [25] and affine-based [27, 28] neighborhoods or Delaunay triangulation [32, 28] have been employed. In most cases, these approaches require an initial set of robust seed matches for initialization whose selection may become critical [74], while other approaches explicitly exploit descriptor similarities to rank matches [32]. In this respect, the vanilla RANSAC and GMS are pure geometric image filters. Notice that descriptor similarities have been also exploited successfully within RANSAC [75]. Closely connected to local filters, another class of methods is designed instead to estimate and interpolate the motion field upon which to check match consistency [29, 31, 30]. Local neighborhood consistency can be framed also into RANSAC as done by Adaptive Locally-Affine Matching (AdaLAM) [33] which runs multiple local affine RANSACs inside the circular neighborhoods of seed matches. AdaLAM design can be related with GroupSAC [76], which draws hypothesis model sample correspondences from different clusters, and with the more general multi-model fitting problem for which several solutions have been investigated [77, 78]. Among these approaches the recent Consensus Clustering (CC) [79] has been developed for the specific case of multiple planar homographies.
RANSAC 利用了全局几何约束。比 RANSAC 更为普遍且宽松的几何假设是仅考虑图像间的局部邻域一致性，如基于网格的运动统计（GMS）[26]采用方形邻域，但也有使用圆形[25]、仿射[27, 28]邻域或德劳内三角剖分[32, 28]的方法。在大多数情况下，这些方法需要一组鲁棒的初始匹配种子进行初始化，其选择可能变得至关重要[74]，而其他方法则明确利用描述符相似性对匹配进行排序[32]。就此而言，原始的 RANSAC 和 GMS 是纯粹的几何图像滤波器。值得注意的是，描述符相似性在 RANSAC 内部也得到了成功利用[75]。与局部滤波器紧密相关的是，另一类方法旨在估计并插值运动场，以此检查匹配一致性[29, 31, 30]。局部邻域一致性也可以纳入 RANSAC 框架，如自适应局部仿射匹配（AdaLAM）[33]在种子匹配的圆形邻域内运行多个局部仿射 RANSAC。 AdaLAM 设计可与 GroupSAC[76]相关联，后者从不同聚类中提取假设模型样本对应关系，并与更普遍的多模型拟合问题相关，对此已有多种解决方案被探讨[77, 78]。在这些方法中，最新的共识聚类（CC）[79]针对多平面单应性的特定情况进行了开发。

Deep learning evolution has provided a boost to image matching as well as other computer vision research fields thanks to the evolution of the deep architectures, hardware and software computational capability and design, and the huge amount of datasets now available for the training. Hybrid pipelines are raised first, replacing handcrafted descriptors with deep ones, as the feature descriptor module is the most natural and immediate to be replaced in the pipeline evolution. SOTA in this sense is represented by the Hard Network (HardNet) [80] and similar networks [81, 82] trained on contrastive learning with hard mining. For what concerns the keypoint extraction, better results with respect to the handcrafted counterparts were achieved by driving the network to mimic handcrafted detector design as for Key.Net [52]. Deep learning has been successfully applied to design standalone patch normalization modules to estimate the reference patch orientation [83, 84] or more complete affine transformations as in the Affine Network [84]. The mentioned deep modules have been employed successfully in many hybrid matching pipelines, especially in conjunction with SIFT [11] or Harris corners [85].
深度学习的演进为图像匹配及其他计算机视觉研究领域带来了显著提升，这得益于深度架构、硬件与软件计算能力及设计的进步，以及如今可用于训练的海量数据集。首先提出的混合管道方案，以深度特征描述符取代手工设计的描述符，因为特征描述模块在管道演进中最自然且直接可被替换。在此背景下，最先进的代表是 HardNet（HardNet）[80]及其类似网络[81, 82]，它们通过对比学习与硬挖掘进行训练。在关键点提取方面，通过引导网络模仿手工检测器设计，如 Key.Net [52]，取得了优于手工设计方法的更好结果。深度学习还被成功应用于设计独立的图像块归一化模块，以估计参考图像块的方向[83, 84]，或更复杂的仿射变换，如 AffineNet [84]中所实现。 上述深度模块已在众多混合匹配流程中成功应用，尤其在与 SIFT[11]或 Harris 角点[85]结合时表现出色。

Full end-to-end deep image matching architectures arose with SuperGlue [94], which incorporates a proper matching network exploiting self and cross attentions through transformers to jointly encode keypoint positions and visual data into a graph structure, and a differentiable form of the Sinkhorn algorithm to provide the final assignments. SuperGlue can represent a breakthrough in the areas and has been extended by the recent LightGlue [12] improving its efficiency, accuracy and training process. Detector-free end-to-end image matching architectures have been also proposed, which avoid explicitly computing a sparse keypoint map, considering instead a semi-dense strategy. The Local Feature TRansformer (LoFTR) [14] is a competitive realization of these matching architectures, which deploys a coarse-to-fine strategy using self and cross attention layers on dense feature maps and has been employed as a base for further design extensions [15, 16]. Notice that end-to-end image matching networks are not fully rotational invariant and can handle only relatively small scene rotations, except specific architectures designed with this purpose [95, 96, 97]. Coarse-to-fine strategy following LoFTR design has been also applied to hybrid canonical pipelines with only deep patch and descriptor modules, leading to matching results comparable in challenging scenarios at an increased computation cost [50]. More recently, dense image matching networks that directly estimate the scene as dense point maps [17] or employing Gaussian process [18], achieve SOTA results in many benchmarks but are more computationally expensive than sparse or semi-dense methods.
全端到端深度图像匹配架构随着 SuperGlue[94]的出现而兴起，该架构通过利用自注意力和交叉注意力机制的匹配网络，将关键点位置和视觉数据联合编码为图结构，并采用可微分的 Sinkhorn 算法进行最终分配。SuperGlue 在相关领域代表了突破性进展，并由最近的 LightGlue[12]进一步扩展，提升了其效率、准确性和训练过程。此外，还提出了无需显式计算稀疏关键点图的端到端图像匹配架构，转而采用半密集策略。Local Feature TRansformer (LoFTR)[14]是这些匹配架构的竞争性实现，它采用粗到细的策略，在密集特征图上使用自注意力和交叉注意力层，并已被用作进一步设计扩展的基础[15, 16]。值得注意的是，端到端图像匹配网络并非完全旋转不变，仅能处理相对较小的场景旋转，除非是为此目的设计的特定架构[95, 96, 97]。 LoFTR 设计中的由粗到细策略也被应用于仅包含深度块和描述符模块的混合规范管道，在计算成本增加的情况下，在具有挑战性的场景中实现了可比的匹配结果[50]。最近，直接估计场景为密集点图[17]或采用高斯过程[18]的密集图像匹配网络，在许多基准测试中达到了最先进的结果，但相比于稀疏或半密集方法，其计算成本更高。

The first effective deep matching filter module was achieved with the introduction of the context normalization [34], which is the normalization by mean and standard deviation, that guarantees to preserve permutation equivalence. The Attentive Context Network (ACNe) [35] further includes local and global attention to exclude outliers from context normalization, while the Order-Aware Network (OANet) [36] adds additional layers to learn how to cluster unordered sets of correspondences to incorporate the data context and the spatial correlation. The Consensus Learning Network (CLNet) [37] prunes matches by filtering data according a local-to-global dynamic neighborhood consensus graphs in consecutive rounds. The Neighbor Consistency Mining Network (NCMNet) [38] improves CLNet architecture by considering different neighborhoods in the feature space, the keypoint coordinate space, and a further global space that considers both previous ones altogether with cross-attention. More recently, the Multiple Sparse Semantics Dynamic Graph Network (MS²DG-Net) [39] designs the filtering in terms of a neighborhood graph through transformers [98] to capture semantics similar structures in the local topology among correspondences. Unlike previous deep approaches, ConvMatch[40] builds a smooth motion field by making use of Convolutional Neural Network (CNN) layers to verify the consistency of the matches. DeMatch [41] refines ConvMatch motion field estimation for a better accommodation of scene disparities by decomposing the rough motion field into several sub-fields. Finally, notice that deep differentiable RANSAC modules have been investigated as well [99, 100].
首个有效的深度匹配滤波模块通过引入上下文归一化[34]得以实现，该归一化基于均值和标准差，确保了置换等价性。注意力上下文网络（ACNe）[35]进一步结合局部和全局注意力，以排除上下文归一化中的异常值，而顺序感知网络（OANet）[36]则增加了额外层，用于学习如何聚类无序的对应集，以整合数据上下文和空间相关性。共识学习网络（CLNet）[37]通过在连续轮次中依据局部到全局的动态邻域共识图过滤数据来修剪匹配。邻域一致性挖掘网络（NCMNet）[38]通过考虑特征空间、关键点坐标空间以及综合前两者的全局空间中的不同邻域，改进了 CLNet 架构，并引入了交叉注意力机制。 最近，多重稀疏语义动态图网络（MS ² DG-Net）[39]通过变换器[98]设计了基于邻域图的过滤机制，以捕捉对应关系间局部拓扑中的语义相似结构。与先前的深度方法不同，ConvMatch[40]利用卷积神经网络（CNN）层构建平滑的运动场，以验证匹配的一致性。DeMatch[41]则通过将粗糙的运动场分解为多个子场，改进了 ConvMatch 的运动场估计，从而更好地适应场景差异。最后，值得注意的是，深度可微分的 RANSAC 模块也得到了研究[99, 100]。

The coarse-to-fine strategy trend has pushed the research of solutions focusing on the refinement of keypoint localization as a standalone module after the coarse assignment of the matches. Worth mentioning is Pixel-Perfect SfM [101] that learns to refine multi-view tracks on their photometric appearance and the SfM structure, while Patch2Pix [42] refines and filters patches exploiting image dense features extracted by the network backbone. Similar to Patch2Pix, Keypt2Subpx [43] is a recent lightweight module to refine only matches requiring as input the descriptors of the local heat map of the related correspondences beside keypoints position. Finally, the Filtering and Calibrating Graph Neural Network (FC-GNN) [44] is an attention-based Graph Neural Network (GNN) jointly leveraging contextual and local information to both filter and refine matches. Unlike Patch2Pix and Keypt2Subpx, does not require feature maps or descriptor confidences as input, as also the proposed MOP+MiHo+NCC. Similarly to FC-GNN, the proposed approach uses both global context and local information to refine matches, respectively to warp and check photometric consistency, but differently from FC-GNN it currently only employs global contextual data since does not implement any feedback system to discard matches on patch similarity after the alignment refinement.
粗到细策略的趋势推动了将关键点定位细化作为粗匹配分配后独立模块的研究。值得一提的是 Pixel-Perfect SfM [101]，它学习根据光度外观和 SfM 结构来细化多视图轨迹，而 Patch2Pix [42]则利用网络主干提取的图像密集特征来细化并过滤补丁。与 Patch2Pix 类似，Keypt2Subpx [43]是一个近期轻量级模块，仅对需要细化的匹配进行处理，输入包括相关对应关系的局部热图描述符以及关键点位置。最后，过滤与校准图神经网络（FC-GNN）[44]是一种基于注意力的图神经网络（GNN），联合利用上下文和局部信息来过滤和细化匹配。与 Patch2Pix 和 Keypt2Subpx 不同，FC-GNN 不要求输入特征图或描述符置信度，正如所提出的 MOP+MiHo+NCC 方法一样。 与 FC-GNN 类似，所提出的方法同时利用全局上下文和局部信息来优化匹配，分别用于扭曲和对光度一致性进行检查，但与 FC-GNN 不同的是，它目前仅采用全局上下文数据，因为尚未实现任何反馈系统在对齐优化后基于块相似性丢弃匹配。

MOP assumes that motion flow within the images can be piecewise approximated by local planar homography.
MOP 假设图像内的运动流可以通过局部平面单应性进行分段近似。

Representing with an abuse of notation the reprojection of a point $\mathbf{x}\in\mathbb{R}^{2}$ through a planar homography $\mathrm{H}\in\mathbb{R}^{3\times 3}$ with $\mathrm{H}$ non-singular as $\mathrm{H}\mathbf{x}$, a match $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ within two keypoints $\mathbf{x}_{1}\in I_{1}$ and $\mathbf{x}_{2}\in I_{2}$ is considered an inlier based on the maximum reprojection error
用符号 $\mathrm{H}\mathbf{x}$ 表示通过非奇异平面单应性 $\mathrm{H}$ 对点 $\mathbf{x}\in\mathbb{R}^{2}$ 的重投影，若两个关键点 $\mathbf{x}_{1}\in I_{1}$ 和 $\mathbf{x}_{2}\in I_{2}$ 之间的匹配 $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ 基于最大重投影误差被视为内点

For a set of matches $\mathcal{M}=\{m\}$, the inlier subset is provided according to a threshold $t$ as
对于一组匹配 $\mathcal{M}=\{m\}$ ，根据阈值 $t$ 提供内点子集

A proper scene planar transformation must be quasi-affine to preserve the convex hull [45] with respect to the minimum sampled model generating $\mathrm{H}$ in RANSAC so the inlier set effectively employed is
正确的场景平面变换必须保持准仿射性，以保留相对于 RANSAC 中最小采样模型生成的凸包，从而使有效利用的内点集为

where $\mathcal{Q}^{\mathcal{M}}_{\mathrm{H}}$ is the set of matches in $\mathcal{M}$ satisfying the quasi-affininity property with respect to the RANSAC minimum model of $\mathrm{H}$ described later in Eq. 12.
其中 $\mathcal{Q}^{\mathcal{M}}_{\mathrm{H}}$ 是 $\mathcal{M}$ 中满足与后文公式 12 所述的 $\mathrm{H}$ 的 RANSAC 最小模型相关的准亲和性属性的匹配集合。

MOP iterates through a loop where a new homography is discovered and the match set is reduced for the next iteration. Iterations halt when the counter $c_{f}$ of the sequential failures at the end of a cycle reaches the maximum allowable value $c_{f}^{\star}=3$. If not incremented at the end of the cycle, $c_{f}$ is reset to 0. As output, MOP returns the set $\mathcal{H}^{\star}$ of found homographies $\mathrm{H}$ which covers the image visual flow.
MOP 通过循环迭代，每次发现新的单应性矩阵并缩减匹配集以进行下一次迭代。当循环结束时连续失败的计数器 $c_{f}$ 达到最大允许值 $c_{f}^{\star}=3$ 时，迭代停止。如果在循环结束时未增加， $c_{f}$ 将被重置为 0。作为输出，MOP 返回一组 $\mathcal{H}^{\star}$ 已找到的单应性矩阵 $\mathrm{H}$ ，这些矩阵覆盖了图像视觉流。

Starting with $c_{f}=0$, $\mathcal{H}^{0}=\emptyset$ and the initial set of input matches $\mathcal{M}^{0}$ provided by the image matching pipeline, MOP at each iteration $k$ looks for the best planar homography $\mathrm{H}^{k}$ compatible with the current match set $\mathcal{M}^{k}$ according to a relaxed inlier threshold $t_{l}=15$ px. When the number of inliers provided by $t_{l}$ is less than the minimum required amount, set to $n=12$, i.e.
从 $c_{f}=0$ 、 $\mathcal{H}^{0}=\emptyset$ 以及图像匹配流水线提供的初始输入匹配集 $\mathcal{M}^{0}$ 开始，MOP 在每次迭代 $k$ 中寻找与当前匹配集 $\mathcal{M}^{k}$ 最兼容的最佳平面单应性 $\mathrm{H}^{k}$ ，依据的是一个放宽的内点阈值 $t_{l}=15$ 像素。当 $t_{l}$ 提供的内点数量少于所需的最小数量 $n=12$ 时，即

the current $\mathrm{H}^{k}$ homography is discarded and the failure counter $c_{f}$ is incremented without updating the iteration counter $k$, the next match set $\mathcal{M}^{k+1}$ and the output list $\mathcal{H}^{k+1}$. Otherwise, MOP removes the inlier matches of $\mathrm{H}^{k}$ for a threshold $t^{k}$ for the next iteration $k+1$
当前的 $\mathrm{H}^{k}$ 单应性被丢弃，失败计数器 $c_{f}$ 递增，而不更新迭代计数器 $k$ 、下一匹配集 $\mathcal{M}^{k+1}$ 和输出列表 $\mathcal{H}^{k+1}$ 。否则，MOP 将移除 $\mathrm{H}^{k}$ 的内点匹配，以 $t^{k}$ 为阈值，用于下一次迭代 $k+1$ 。

where $\mathcal{I}^{k}=\mathcal{I}^{\mathcal{M}^{k}}_{\mathrm{H}^{k},t^{k}}$ and updates the output list
其中 $\mathcal{I}^{k}=\mathcal{I}^{\mathcal{M}^{k}}_{\mathrm{H}^{k},t^{k}}$ 并更新输出列表

The threshold $t^{k}$ adapts to the evolution of the homography search and is equal to a strict threshold $t_{h}=t_{l}/2$ in case the cardinality of the inlier set using $t_{h}$ is greater than half of $n$, i.e. in the worst case no more than half of the added matches should belong to previous overlapping planes. If this condition does not hold $t^{k}$ turns into the relaxed threshold $t_{l}$ and the failure counter $c_{f}$ is incremented
阈值 $t^{k}$ 适应单应性搜索的演变，并且在使用 $t_{h}$ 的内点集基数大于 $n$ 的一半时，等于严格的阈值 $t_{h}=t_{l}/2$ ，即在最坏情况下，不超过一半的添加匹配应属于先前的重叠平面。如果此条件不成立， $t^{k}$ 变为宽松的阈值 $t_{l}$ ，并且失败计数器 $c_{f}$ 递增。

Using the relaxed threshold $t_{l}$ to select the best homography $\mathrm{H}^{k}$ is required since the real motion flow is only locally approximated by planes, and removing only strong inliers by $t_{h}$ for the next iteration $k+1$ guarantees smooth changes within overlapping planes and more robustness to noise. Nevertheless, in case of slow convergence or when the homography search gets stuck around a wide overlapping or noisy planar configuration, limiting the search to matches excluded by previous planes can provide a way out. The final set of homography returned is
使用松弛阈值 $t_{l}$ 来选择最佳单应性 $\mathrm{H}^{k}$ 是必要的，因为实际运动流仅由局部平面近似，并且仅通过 $t_{h}$ 移除强内点以进行下一次迭代 $k+1$ ，可确保重叠平面内的平滑变化并增强对噪声的鲁棒性。然而，在收敛缓慢或单应性搜索陷入广泛重叠或噪声平面配置的情况下，将搜索限制在先前平面排除的匹配点上，可提供一种解决方案。最终返回的单应性集合为

found by the last iteration $k^{\star}$ when the failure counter reaches $c_{f}^{\star}$.
由最后一次迭代 $k^{\star}$ 发现，当失败计数器达到 $c_{f}^{\star}$ 时。

Within each MOP iteration $k$, a homography $\mathrm{H}^{k}$ is extracted by RANSAC on the match set $\mathcal{M}^{k}$. The vanilla RANSAC is modified according to the following heuristics to improve robustness and avoid degenerate cases.
在每个 MOP 迭代 $k$ 中，通过在匹配集 $\mathcal{M}^{k}$ 上应用 RANSAC 提取单应性 $\mathrm{H}^{k}$ 。根据以下启发式方法对传统 RANSAC 进行修改，以提高鲁棒性并避免退化情况。

A minimum number $c_{\min}=50$ of RANSAC loop iterations is required besides the max number of iterations $c_{\max}=2000$. In addition, a minimum hypothesis sampled model
除了最大迭代次数 $c_{\max}=2000$ 外，还需要最少的 RANSAC 循环迭代次数 $c_{\min}=50$ 。此外，还需要采样模型的最小假设。

is discarded if the distance within the keypoints in one of the images is less than the relaxed threshold $t_{l}$, i.e. it must hold
如果图像中关键点之间的距离小于松弛阈值 $t_{l}$ ，则被舍弃，即必须满足

Furthermore, being $\mathrm{H}$ the corresponding homography derived by the normalized 8-point algorithm [45], the smaller singular value found by solving through SVD the $8\times 9$ homogeneous system must be relatively greater than zero for a stable solution not close to degeneracy. According to these observations, the minimum singular value is constrained to be greater than 0.05. Note that an absolute threshold is feasible since data are normalized before SVD.
此外，由于 $\mathrm{H}$ 对应于通过归一化 8 点算法[45]推导出的单应性，通过 SVD 求解 $8\times 9$ 齐次系统时发现的最小奇异值必须相对大于零，以确保解的稳定性，避免接近退化。基于这些观察，最小奇异值被限制在大于 0.05。需要注意的是，由于 SVD 之前数据已归一化，因此采用绝对阈值是可行的。

Next, matches in $\mathcal{S}$ must satisfy quasi-affinity with respect to the derived homography $\mathrm{H}$. This can be verified by checking that the sign of the last coordinate of the reprojection $\mathrm{H}\mathbf{s}_{1i}$ in non-normalized homogenous coordinates are the same for $1\leq i\leq 4$, i.e. for all the four keypoint in the first image. This can be expressed as
接下来， $\mathcal{S}$ 中的匹配点必须满足相对于推导出的单应性矩阵 $\mathrm{H}$ 的准亲和性。这可以通过检查重投影 $\mathrm{H}\mathbf{s}_{1i}$ 在非归一化齐次坐标中的最后一个坐标的符号是否对 $1\leq i\leq 4$ 一致来验证，即对于第一张图像中的所有四个关键点。这可以表示为

where $[\mathbf{x}]_{3}$ denotes the third and last vector element of the point $\mathbf{x}$ in non-normalized homogeneous coordinates. Following the above discussion, the quasi-affine set of matches $\mathcal{Q}^{\mathcal{M}}_{\mathrm{H}}$ in Eq. 3 is formally defined as
其中 $[\mathbf{x}]_{3}$ 表示点 $\mathbf{x}$ 在非归一化齐次坐标中的第三个也是最后一个向量元素。根据上述讨论，匹配的准仿射集 $\mathcal{Q}^{\mathcal{M}}_{\mathrm{H}}$ 在公式 3 中正式定义为

For better numerical stability, the analogous checks are executed also in the reverse direction through $\mathrm{H}^{-1}_{\mathcal{S}}$.
为了提高数值稳定性，类似的检查也在 $\mathrm{H}^{-1}_{\mathcal{S}}$ 中以反向执行。

Lastly, to speed up the RANSAC search, a buffer $\mathcal{B}=\{\tilde{\mathrm{H}}_{i}\}$ with $1\leq i\leq z$ is globally maintained in order to contain the top $z=5$ discarded sub-optimal homographies encountered within a RANSAC run. For the current match set $\mathcal{M}^{k}$ if $\tilde{\mathrm{H}}_{0}$ is the current best RANSAC model maximizing the number of inliers, $\tilde{\mathrm{H}}_{i}\in\mathcal{B}$ are ordered by the number $v_{i}$ of inliers excluding those compatible with the previous homographies in the buffer, i.e. it must holds that
最后，为了加速 RANSAC 搜索，全局维护了一个缓冲区 $\mathcal{B}=\{\tilde{\mathrm{H}}_{i}\}$ ，其中包含 $1\leq i\leq z$ ，用于存储在一次 RANSAC 运行中遇到的排名前 $z=5$ 的次优单应性矩阵。对于当前的匹配集 $\mathcal{M}^{k}$ ，如果 $\tilde{\mathrm{H}}_{0}$ 是当前最佳 RANSAC 模型，其内点数量最多，那么 $\tilde{\mathrm{H}}_{i}\in\mathcal{B}$ 则按内点数量 $v_{i}$ 排序，排序时排除那些与缓冲区中先前单应性矩阵兼容的内点，即必须满足以下条件：

where 其中

The buffer $\mathcal{B}$ can be updated when the number of inliers of the homography associated with the current sampled hypothesis is greater than the minimum $v_{i}$, by inserting in the case the new homography, removing the low-scoring one and resorting to the buffer. Moreover, at the beginning of a RANSAC run inside a MOP iteration, the minimum hypothesis model sampling sets $\mathcal{S}$ corresponding to homographies in $\mathcal{B}$ are evaluated before proceeding with true random samples to provide a bootstrap for the solution search. This also guarantees a global sequential inspection of the overall solution search space within MOP.
缓冲区 $\mathcal{B}$ 可以在与当前采样假设相关联的单应性的内点数量大于最小值 $v_{i}$ 时进行更新，具体操作是插入新的单应性，移除得分较低的单应性，并重新调整缓冲区。此外，在 MOP 迭代内部的 RANSAC 运行开始时，会先评估与 $\mathcal{B}$ 中单应性对应的假设模型采样集 $\mathcal{S}$ ，然后再进行真正的随机采样，以此为解搜索提供引导。这同时也确保了在 MOP 内对整个解搜索空间进行全局顺序检查。

After the final set of homographies $\mathcal{H}^{\star}$ is computed, the filtered match set $\mathcal{M}^{\star}$ is obtained by removing all matches $m\in\mathcal{M}$ for which there is no homography $\mathrm{H}$ in $\mathcal{H}^{\star}$ having them as inlier, i.e.
在计算出最终的一组单应性矩阵 $\mathcal{H}^{\star}$ 后，通过移除所有匹配项 $m\in\mathcal{M}$ 来获得过滤后的匹配集 $\mathcal{M}^{\star}$ ，这些匹配项 $m\in\mathcal{M}$ 在 $\mathcal{H}^{\star}$ 中没有对应的单应性矩阵 $\mathrm{H}$ 将其视为内点，即

Next, a homography must be associated with each survived match $m\in\mathcal{M}^{\star}$, which will be used for patch normalization.
接下来，必须为每个幸存的匹配项 $m\in\mathcal{M}^{\star}$ 关联一个单应性矩阵，用于块归一化。

A possible choice is to assign the homography $\mathrm{H}_{m}^{\varepsilon}$ which gives the minimum reprojection error, i.e.
一个可能的选择是分配单应性矩阵 $\mathrm{H}_{m}^{\varepsilon}$ ，它产生最小的重投影误差，即

However, this choice is likely to select homographies corresponding to narrow planes in the image with small consensus sets, which leads to a low reprojection error but more unstable and prone to noise assignment.
然而，这种选择可能会挑选出对应于图像中狭窄平面的单应性变换，这些变换具有较小的共识集，从而导致重投影误差较低，但更不稳定且易受噪声影响。

Another choice is to assign to the match $m$ the homography $\mathrm{H}_{m}^{\mathcal{I}}$ compatible with $m$ with the wider consensus set, i.e.
另一种选择是将匹配 $m$ 分配给与 $m$ 兼容且具有更广泛共识集的单应性 $\mathrm{H}_{m}^{\mathcal{I}}$ ，即

which points to more stable and robust homographies but can lead to incorrect patches when the corresponding reprojection error is not among the minimum ones.
这表明了更稳定和鲁棒的单应性，但在对应的重投影误差并非最小值时，可能导致错误的图像块。

The homography $\mathrm{H}_{m}$ actually chosen for the match association provides a solution within the above. Specifically, for a match $m$, defining $q_{m}$ as the median of the number of inliers of the top 5 compatible homographies according to the number of inliers for $t_{l}$, $\mathrm{H}_{m}$ is the compatible homography with the minimum reprojection error among those with an inlier number at least equal to $q_{m}$
所选用于匹配关联的单应性矩阵 $\mathrm{H}_{m}$ 确实在上述范围内提供了解决方案。具体而言，对于一个匹配 $m$ ，定义 $q_{m}$ 为根据内点数量对前 5 个兼容单应性矩阵的内点数量中位数， $\mathrm{H}_{m}$ 是在内点数量至少等于 $q_{m}$ 的单应性矩阵中，具有最小重投影误差的兼容单应性矩阵。

Figure 1 shows an example of the achieved solution directly in combination with MiHo homography estimation described hereafter in Sec. 3.2, since MOP or MOP+MiHo in visual cluster representation does not present appreciable differences. Keypoint belonging to discarded matches are marked with black diamonds, while the clusters highlighted by other combinations of markers and colors indicate the resulting filtered matches $m\in\mathcal{M}^{\star}$ with the selected planar homography $\mathrm{H}_{m}$. Notice that clusters are not designed to highlight scene planes but points that move according to the same homography transformation within the image pair.
图 1 展示了一个直接结合 MiHo 单应性估计（详见第 3.2 节）所获得的解决方案示例，因为在视觉聚类表示中，MOP 或 MOP+MiHo 并未呈现显著差异。属于被丢弃匹配的关键点以黑色菱形标记，而其他组合标记和颜色高亮的聚类则表示经过筛选的匹配结果 $m\in\mathcal{M}^{\star}$ ，以及选定的平面单应性 $\mathrm{H}_{m}$ 。值得注意的是，聚类并非旨在突出场景平面，而是强调在图像对中根据相同单应性变换移动的点。

The $\mathrm{H}_{m}$ homography estimated by Eq. 18 for the match $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ allows to reproject the local patch neighborhood of $x_{1}\in I_{1}$ onto the corresponding one centered in $\mathbf{x}_{2}\in I_{2}$. Nevertheless, $\mathrm{H}_{m}$ approximates the true transformation within the images which becomes less accurate as long as the distance from the keypoint center increases. This implies that patch alignment could be invalid for wider, and theoretically more discriminant, patches.
通过公式 18 估计的匹配 $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ 的 $\mathrm{H}_{m}$ 单应性，可以将 $x_{1}\in I_{1}$ 的局部块邻域重新投影到以 $\mathbf{x}_{2}\in I_{2}$ 为中心的对应邻域上。然而， $\mathrm{H}_{m}$ 近似于图像内的真实变换，随着与关键点中心距离的增加，其准确性逐渐降低。这意味着对于更宽、理论上更具辨别力的块，块对齐可能失效。

The idea behind MiHo is to break the homography $\mathrm{H}_{m}$ in the middle by two homographies $\mathrm{H}_{m_{1}}$ and $\mathrm{H}_{m_{2}}$ such that $\mathrm{H}_{m}=\mathrm{H}_{m_{2}}\mathrm{H}_{m_{1}}$ so that the patch on $\mathbf{x}_{1}$ gets deformed by $\mathrm{H}_{m_{1}}$ less than $\mathrm{H}_{m}$, and likewise the patch on $\mathbf{x}_{2}$ gets less deformed by $\mathrm{H}_{m_{2}}^{-1}$ than $\mathrm{H}_{m}^{-1}$. This means that visually on the reprojected patch a unit area square should remain almost similar to the original one. As this must hold from both the images, the deformation error must be distributed almost equally between the two homographies $\mathrm{H}_{m_{1}}$ and $\mathrm{H}_{m_{2}}$. Moreover, since interpolation degrades with up-sampling, MiHo aims to balance the down-sampling of the patch at finer resolution and the up-sampling of the patch at the coarser resolution.
MiHo 背后的理念是通过两个单应性变换 $\mathrm{H}_{m_{1}}$ 和 $\mathrm{H}_{m_{2}}$ 将单应性 $\mathrm{H}_{m}$ 在中间打破，使得 $\mathrm{H}_{m}=\mathrm{H}_{m_{2}}\mathrm{H}_{m_{1}}$ ，从而使得位于 $\mathbf{x}_{1}$ 上的图像块经 $\mathrm{H}_{m_{1}}$ 变形程度小于 $\mathrm{H}_{m}$ ，同样地，位于 $\mathbf{x}_{2}$ 上的图像块经 $\mathrm{H}_{m_{2}}^{-1}$ 变形程度也小于 $\mathrm{H}_{m}^{-1}$ 。这意味着在重新投影的图像块上，单位面积的正方形应与原始形状几乎相似。由于这一要求必须从两幅图像中都成立，变形误差必须在两个单应性 $\mathrm{H}_{m_{1}}$ 和 $\mathrm{H}_{m_{2}}$ 之间几乎均匀分布。此外，鉴于插值质量随上采样而下降，MiHo 旨在平衡在更精细分辨率下对图像块的下采样与在较粗分辨率下对图像块的上采样。

While a strict analytical formulation of the above problem which leads to a practical satisfying solution is not easy to derive, the heuristic found in [51] can be exploited to modify RANSAC within MOP to account for the required constraints. Specifically, each match $m$ is replaced by two corresponding matches $m_{1}=(\mathbf{x}_{1},\mathbf{m})$ and $m_{2}=(\mathbf{m},\mathbf{x}_{2})$ where $\mathbf{m}$ is the midpoint within the two keypoints
虽然严格分析上述问题并导出一个实际满意的解决方案并不容易，但[51]中发现的启发式方法可以被利用来修改 MOP 中的 RANSAC，以考虑所需约束。具体而言，每个匹配 $m$ 被替换为两个对应的匹配 $m_{1}=(\mathbf{x}_{1},\mathbf{m})$ 和 $m_{2}=(\mathbf{m},\mathbf{x}_{2})$ ，其中 $\mathbf{m}$ 是两个关键点之间的中点。

Hence the RANSAC input match set $\mathcal{M}^{k}=\{m\}$ for the MOP $k$-th iteration is split in the two match sets $\mathcal{M}^{k}_{1}=\{m_{1}\}$ and $\mathcal{M}^{k}_{2}=\{m_{2}\}$. Within a RANSAC iteration, a sample $\mathcal{S}$ defined by Eq. 9 is likewise split into
因此，MOP 第 $k$ 次迭代的 RANSAC 输入匹配集 $\mathcal{M}^{k}=\{m\}$ 被划分为两个匹配集 $\mathcal{M}^{k}_{1}=\{m_{1}\}$ 和 $\mathcal{M}^{k}_{2}=\{m_{2}\}$ 。在 RANSAC 迭代过程中，由公式 9 定义的样本 $\mathcal{S}$ 同样被分割为

leading to two concurrent homographies $\mathrm{H}_{1}$ and $\mathrm{H}_{2}$ to be verified simultaneously with the inlier set in analogous way to Eq. 3
导致两个并发的单应性 $\mathrm{H}_{1}$ 和 $\mathrm{H}_{2}$ 需与内点集以类似式 3 的方式同时验证

where for two generic match set $\mathcal{M}_{1}$, $\mathcal{M}_{2}$, the $\upharpoonleft\downharpoonright$ operator rejoints splitted matches according to
其中，对于两个通用的匹配集 $\mathcal{M}_{1}$ 、 $\mathcal{M}_{2}$ ， $\upharpoonleft\downharpoonright$ 运算符根据规则重新连接已拆分的匹配项

All the other MOP steps defined in Sec. 3.1, including RANSAC degeneracy checks, threshold adaptation, and final homography assignment, follow straightly in an analogous way. Overall, the principal difference is that a pair of homography $(\mathrm{H}_{m_{1}},\mathrm{H}_{m_{2}})$ is obtained instead of the single $\mathrm{H}_{m}$ homography to be operated simultaneously on split match sets $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ instead of the original one $\mathcal{M}$.
第 3.1 节中定义的所有其他 MOP 步骤，包括 RANSAC 退化检查、阈值调整和最终单应性分配，均以类似方式直接进行。总体而言，主要区别在于获得的是一对单应性 $(\mathrm{H}_{m_{1}},\mathrm{H}_{m_{2}})$ ，而非单一的 $\mathrm{H}_{m}$ 单应性，同时作用于分割匹配集 $\mathcal{M}_{1}$ 和 $\mathcal{M}_{2}$ ，而非原始的 $\mathcal{M}$ 。

Figure 2 shows the differences in applying MiHo to align to planar scenes with respect to directly reproject in any of the two images considered as reference. It can be noted that the distortion differences are overall reduced, as highlighted by the corresponding grid deformation. Moreover, MiHo homography strongly resembles an affine transformation or even a similarity, i.e. a rotation and scale change only, imposing in some sense a tight constraint on the transformation than the base homography. To balance this additional restrain with respect to the original MOP, in MOP+MiHo the minimum requirement amount of inliers for increasing the failure counter defined by Eq. 4 has been relaxed after experimental validation from $n=12$ to $n=8$.
图 2 展示了将 MiHo 应用于对齐平面场景与直接在任意两张被视为参考的图像中重新投影之间的差异。可以看出，扭曲差异整体上有所减少，这在相应的网格变形中得到了体现。此外，MiHo 单应性非常接近仿射变换，甚至相似变换，即仅涉及旋转和尺度变化，在某种意义上对变换施加了比基础单应性更严格的约束。为了在 MOP+MiHo 中平衡这一额外约束与原始 MOP 的关系，经过实验验证后，增加失败计数器所需的最小内点数量已从 $n=12$ 放宽至 $n=8$ 。

MiHo is invariant to translations as long as the algorithm used to extract the homography provides the same invariance. This can be easily verified since when translation vectors $\mathbf{t}_{a}$, $\mathbf{t}_{b}$ are respectively added to all keypoints $\mathbf{x}_{1}\in I_{1}$ and $\mathbf{x}_{2}\in I_{2}$ to get $\mathbf{x}^{\prime}_{1}=\mathbf{x}_{1}+\mathbf{t}_{a}$ and $\mathbf{x}^{\prime}_{2}=\mathbf{x}_{2}+\mathbf{t}_{b}$ the corresponding midpoints are of the form
只要用于提取单应性的算法提供相同的平移不变性，MiHo 对平移是不变的。这一点很容易验证，因为当平移向量 $\mathbf{t}_{a}$ 和 $\mathbf{t}_{b}$ 分别添加到所有关键点 $\mathbf{x}_{1}\in I_{1}$ 和 $\mathbf{x}_{2}\in I_{2}$ 以得到 $\mathbf{x}^{\prime}_{1}=\mathbf{x}_{1}+\mathbf{t}_{a}$ 和 $\mathbf{x}^{\prime}_{2}=\mathbf{x}_{2}+\mathbf{t}_{b}$ 时，相应的中间点形式为

where $\mathbf{t}=\frac{\mathbf{t}_{a}+\mathbf{t}_{b}}{2}$ is a fixed translation vector is added to all the corresponding original midpoints $\mathbf{m}$. Overall, translation keypoints in the respective images have the effect of adding a fixed translation to the corresponding midpoint. Hence MiHo is invariant to translations since the normalization employed by the normalized 8-point algorithm is invariant to similarity transformations [45], including translations.
其中， $\mathbf{t}=\frac{\mathbf{t}_{a}+\mathbf{t}_{b}}{2}$ 是一个固定的平移向量，被加到所有相应的原始中点 $\mathbf{m}$ 上。总体而言，在各自图像中的平移关键点具有为相应中点添加固定平移的效果。因此，MiHo 对平移是不变的，因为归一化 8 点算法所采用的归一化处理对包括平移在内的相似变换是不变的 [45]。

MiHo is not rotation invariant. Figure 3 illustrates how midpoints change with rotations. In the ideal MiHo configuration where the images are upright the area of the image formed in the midpoint plane is within the areas of the original ones, making a single cone when considering the images as stacked. The area of the midpoint plane is lower than the minimum area between the two images when there is a rotation about $180^{\circ}$, and could degenerate to the apex of the corresponding double cone. A simple strategy to get an almost-optimal configuration was experimentally verified to work well in practice under the observation that images acquired by cameras tend to have a possible relative rotation $\alpha$ of a multiple of $90^{\circ}$.
MiHo 不具有旋转不变性。图 3 展示了中点随旋转的变化情况。在理想的 MiHo 配置中，当图像竖直时，中点平面内形成的图像面积介于原始图像面积之间，将图像视为堆叠时形成一个单锥。当围绕 $180^{\circ}$ 进行旋转时，中点平面的面积小于两图像间的最小面积，并可能退化为对应双锥的顶点。通过实验验证，一种简单的策略在实践中表现良好，该策略基于以下观察：相机获取的图像往往具有 $\alpha$ 的相对旋转，且该旋转为 $90^{\circ}$ 的倍数。

In detail, let us define a given input match $m_{i}=(\mathbf{x}_{1i},\mathbf{x}^{\alpha}_{2i})\in\mathcal{M}^{\alpha}$ and the corresponding midpoint $\mathbf{m}^{\alpha}_{i}=\frac{\mathbf{x}_{1i}+\mathbf{x}_{2i}^{\alpha}}{2}$, where $\mathbf{x}_{1i}\in I_{1}$ and $\mathbf{x}^{\alpha}_{2i}\in I^{\alpha}_{2}$ being the image $I_{2}$ rotated by $\alpha$. One has to choose $\alpha^{\star}\in\mathcal{A}=\{0,\tfrac{\pi}{2},\pi,\tfrac{3\pi}{2}\}$ to maximize the configuration for which midpoint inter-distances are between corresponding keypoint inter-distances
具体而言，我们定义一个给定的输入匹配 $m_{i}=(\mathbf{x}_{1i},\mathbf{x}^{\alpha}_{2i})\in\mathcal{M}^{\alpha}$ 及其对应的中间点 $\mathbf{m}^{\alpha}_{i}=\frac{\mathbf{x}_{1i}+\mathbf{x}_{2i}^{\alpha}}{2}$ ，其中 $\mathbf{x}_{1i}\in I_{1}$ 和 $\mathbf{x}^{\alpha}_{2i}\in I^{\alpha}_{2}$ 是图像 $I_{2}$ 旋转 $\alpha$ 后的结果。需要选择 $\alpha^{\star}\in\mathcal{A}=\{0,\tfrac{\pi}{2},\pi,\tfrac{3\pi}{2}\}$ 以最大化中间点间距在对应关键点间距之间的配置。

where 其中

and $\llbracket\cdot\rrbracket$ is the Iverson bracket. The global orientation estimation $\alpha^{\star}$ can be computed efficiently on the initial input matches $\mathcal{M}$ before running MOP, adjusting keypoints accordingly. Final homographies can be adjusted in turn by removing the rotation given by $\alpha^{\star}$. Figure 4 shows the MOP+MiHo result obtained on a set of matches without or with orientation pre-processing, highlighting the better matches obtained in the latter case.
其中 $\llbracket\cdot\rrbracket$ 是艾弗森括号。全局方向估计 $\alpha^{\star}$ 可以在运行 MOP 之前，通过对初始输入匹配 $\mathcal{M}$ 进行高效计算，并相应调整关键点。最终的单应性矩阵可以通过去除由 $\alpha^{\star}$ 给出的旋转来进行调整。图 4 展示了在未进行或进行方向预处理的情况下，通过 MOP+MiHo 方法在一组匹配上获得的结果，突出了在后一种情况下获得的更好匹配。

NCC is employed to effectively refine corresponding keypoint localization. The approach assumes that corresponding keypoints have been roughly aligned locally by the transformations provided as a homography pair. In the warped aligning space, template matching [48] is employed to refine translation offsets in order to maximize the NCC peak response when the patch of a keypoint is employed as a filter onto the other image of the corresponding keypoint.
NCC 被用于有效精炼相应关键点的定位。该方法假设通过作为单应性对提供的变换，相应关键点已在局部大致对齐。在扭曲对齐的空间中，采用模板匹配[48]来精炼平移偏移，以在将关键点的补丁用作另一图像中对应关键点的滤波器时，最大化 NCC 峰值响应。

For the match $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ within images $I_{1}$ and $I_{2}$ there can be multiple potential choices of the warping homography pair, which are denoted by the set of associated homography pairs $\mathcal{H}_{m}=\{(\mathrm{H}_{1},\mathrm{H}_{2})\}$. In detail, according to the previous steps of the matching pipeline, a candidate is the base homography pair $(\mathrm{H}^{b}_{1},\mathrm{H}^{b}_{2})$ defined as
对于图像 $I_{1}$ 和 $I_{2}$ 中的匹配 $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ ，可能存在多个潜在的扭曲单应性对选择，这些选择由相关单应性对集合 $\mathcal{H}_{m}=\{(\mathrm{H}_{1},\mathrm{H}_{2})\}$ 表示。具体而言，根据匹配流水线的先前步骤，候选者是定义为基单应性对 $(\mathrm{H}^{b}_{1},\mathrm{H}^{b}_{2})$ 的单应性对。

• •

  $(\mathrm{I},\mathrm{I})$, where $\mathrm{I}\in\mathbb{R}^{3\times 3}$ is the identity matrix, when no local neighborhood data are provided, e.g. for SuperPoint;

  Report issue for preceding element
  
  • $(\mathrm{I},\mathrm{I})$ ，其中 $\mathrm{I}\in\mathbb{R}^{3\times 3}$ 是单位矩阵，当未提供局部邻域数据时，例如在 SuperPoint 中；
• •

  $(\mathrm{A}_{1},\mathrm{A}_{2})$, where $\mathrm{A}_{1},\mathrm{A}_{2}\in\mathbb{R}^{3\times 3}$ are affine or similarity transformations in case of affine covariant detector & descriptor, as for AffNet or SIFT, respectively;

  Report issue for preceding element
  
  • $(\mathrm{A}_{1},\mathrm{A}_{2})$ ，其中 $\mathrm{A}_{1},\mathrm{A}_{2}\in\mathbb{R}^{3\times 3}$ 在仿射协变检测器和描述符的情况下分别是仿射或相似变换，如 AffNet 或 SIFT

and the other is extended pair $(\mathrm{H}^{e}_{1},\mathrm{H}^{e}_{2})$ defined as
另一个是扩展对 $(\mathrm{H}^{e}_{1},\mathrm{H}^{e}_{2})$ ，定义为

• •

  $(\mathrm{I},\mathrm{H}_{m})$, where $\mathrm{H}_{m}$ is defined in Eq. 18, in case MOP is used without MiHo;

  Report issue for preceding element
  
  • $(\mathrm{I},\mathrm{H}_{m})$ ，其中 $\mathrm{H}_{m}$ 在公式 18 中定义，适用于未使用 MiHo 的 MOP 情况；
• •

  $(\mathrm{H}_{m_{1}},\mathrm{H}_{m_{2}})$ as described in Sec. 3.2.1, in case of MOP+MiHo

  Report issue for preceding element
  
  • 如 3.2.1 节所述，在 MOP+MiHo 情况下
• •

  $(\mathrm{H}^{b}_{1},\mathrm{H}^{b}_{2})$ as above otherwise.

  Report issue for preceding element
  
  • 如上所述，否则。

Since the found transformations are approximated, the extended homography pair is perturbed in practice by small shear factors $f$ and rotation changes $\rho$ through the affine transformations
由于所发现的变换是近似的，扩展的单应性对在实际应用中通过仿射变换受到微小的剪切因子 $f$ 和旋转变化 $\rho$ 的扰动

with 与

so that 以便

Let us denote a generic image by $I$, the intensity values of $I$ for a generic pixel $\mathbf{x}\in\mathbb{R}^{2}$ as $I(\mathbf{x})$, the window radius extension as
我们用 $I$ 表示一个通用图像， $I$ 在通用像素 $\mathbf{x}\in\mathbb{R}^{2}$ 处的强度值为 $I(\mathbf{x})$ ，窗口半径扩展为

such that the squared window set of offsets is
使得偏移量的平方窗口集为

and has cardinality equal to
且基数等于

The squared patch of $I$ centered in $\mathbf{x}$ with radius $w$ is the pixel set
以 $\mathbf{x}$ 为中心、半径为 $w$ 的 $I$ 方形补丁是像素集

while the mean intensity value of the patch $\mathcal{P}^{I}_{\mathbf{x},r}$ is
而补丁 $\mathcal{P}^{I}_{\mathbf{x},r}$ 的平均强度值为

and the variance is 方差为

The normalized intensity value $\overline{I}_{\mathbf{x},r}(\mathbf{y})$ by the mean and standard deviation of the patch $\mathbf{P}^{I}_{\mathbf{x},r}$ for a pixel $\mathbf{y}$ in $I$ is
通过像素 $\mathbf{y}$ 在 $I$ 中的补丁 $\mathbf{P}^{I}_{\mathbf{x},r}$ 的均值和标准差归一化的强度值 $\overline{I}_{\mathbf{x},r}(\mathbf{y})$ 为

which is ideally robust to affine illumination changes [48]. Lastly, the similarity between two patches $\mathcal{P}^{A}_{\mathbf{a},r}$ and $\mathcal{P}^{B}_{\mathbf{b},r}$ is
这理想地对仿射光照变化具有鲁棒性[48]。最后，两个块 $\mathcal{P}^{A}_{\mathbf{a},r}$ 和 $\mathcal{P}^{B}_{\mathbf{b},r}$ 之间的相似性是

so that, for the match $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ and a set of associated homography pairs $\mathcal{H}_{m}$ derived in Eq. 3.3.1, the refined keypoint offsets $(\mathbf{t}_{1}^{\star},\mathbf{t}_{2}^{\star})$ and the best aligning homography pair $(\mathrm{H}_{1}^{\star},\mathrm{H}_{2}^{\star})$ are given by
从而，对于匹配 $m=(\mathbf{x}_{1},\mathbf{x}_{2})$ 以及在公式 3.3.1 中导出的一组相关单应性对 $\mathcal{H}_{m}$ ，精炼的关键点偏移量 $(\mathbf{t}_{1}^{\star},\mathbf{t}_{2}^{\star})$ 和最佳对齐的单应性对 $(\mathrm{H}_{1}^{\star},\mathrm{H}_{2}^{\star})$ 由以下公式给出：

where $\mathrm{H}\circ I$ means the warp of the image $I$ by $\mathrm{H}$, $\mathrm{H}\mathbf{x}$ is the transformed pixel coordinates and
其中 $\mathrm{H}\circ I$ 表示图像 $I$ 通过 $\mathrm{H}$ 的扭曲， $\mathrm{H}\mathbf{x}$ 是变换后的像素坐标，

is the set of offset translation to check considering in turn one of the images as template since this is not a symmetric process. Notice that $\mathbf{t}^{\star}_{1},\mathbf{t}^{\star}_{2}$ cannot be both different from $\mathbf{0}$ according to $\overleftrightarrow{\mathcal{W}}^{2}_{r}$. The final refined match
是考虑偏移量翻译的集合，依次将其中一张图像作为模板进行检查，因为这是一个非对称过程。注意，根据 $\overleftrightarrow{\mathcal{W}}^{2}_{r}$ ， $\mathbf{t}^{\star}_{1},\mathbf{t}^{\star}_{2}$ 不能同时与 $\mathbf{0}$ 不同。最终的精细匹配

is obtained by reprojecting back to the original images, i.e.
是通过重新投影回原始图像获得的，即

The normalized cross correlation can be computed efficiently by convolution, where bilinear interpolation is employed to warp images. The patch radius $r$ is also the offset search radius and has been experimentally set to $r=10\approx\sqrt{2}t_{h}$ px. According to preliminary experiments, a wider radius extension can break the planar neighborhood assumption and patch alignment.
归一化互相关可以通过卷积高效计算，其中采用双线性插值对图像进行扭曲。补丁半径 $r$ 同时也是偏移搜索半径，并已通过实验设定为 $r=10\approx\sqrt{2}t_{h}$ 像素。根据初步实验，更宽的半径扩展会破坏平面邻域假设及补丁对齐。

The refinement offset can be further enhanced by parabolic interpolation adding sub-pixel precision [102]. Notice that other forms of interpolations have been investigated but they did not provide any sub-pixel accuracy improvements according to previous experiments [49]. The NCC response map at $\mathbf{u}\in\mathbb{R}^{2}$ in the best warped space with origin the peak value can be written as
通过抛物线插值可进一步提升细化偏移量，增加亚像素精度[102]。值得注意的是，尽管其他形式的插值方法已被研究，但根据先前实验[49]，它们并未带来亚像素级别的精度提升。在最佳扭曲空间中，以峰值为原点的 NCC 响应图 $\mathbf{u}\in\mathbb{R}^{2}$ 可表示为

from $T_{m}^{I_{1},I_{2}}$ computed by Eq. 3.3.2. The sub-pixel refinement offsets in the horizontal direction are computed as the vertex $-\tfrac{b}{2a}$ of a parabola $ax^{2}+bx+c=y$ fitted on the horizontal neighborhood centered in the peak of the maximized NNC response map, i.e.
由公式 3.3.2 计算得到的 $T_{m}^{I_{1},I_{2}}$ 。水平方向上的亚像素细化偏移量计算为抛物线 $ax^{2}+bx+c=y$ 的顶点 $-\tfrac{b}{2a}$ ，该抛物线拟合于以最大化 NNC 响应图峰值为中心的水平邻域上，即