Review 评论

Underwater robot sensing technology: A survey
水下机器人传感技术：调查

3.1.1. Underwater Acoustic Ranging/Imaging Sensor
3.1.1.水下声学测距/成像传感器

Underwater acoustic ranging/imaging sensors mainly include single-beam sonar, side-scan sonar, and multibeam sonar.
水下声学测距/成像传感器主要包括单波束声纳、侧扫声纳和 多波束声纳。

Single-Beam Sonar: As shown in Fig. 2(a), the single-beam sonar receives a beam of the short-pulse acoustic signal emitted by a transducer and computes the depth of a submerged object by travel time accordingly. Due to its low cost and easy to use, single-beam sonar is widely used in marine engineering and resource exploration. However, it cannot obtain high-precision measurement results and a wide range of coverage.
单波束声纳：如图 2(a)所示，单波束声纳接收换能器发射的一束短脉冲声学信号，并通过相应的传播时间计算出水下物体的深度。由于单波束声纳成本低、使用方便，被广泛应用于海洋工程和资源勘探。然而，它无法获得高精度的测量结果和大范围的覆盖范围。

1. Download: Download high-res image (124KB)
   下载：下载高清图片 (124KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 2. Typical sonar: (a) single-beam sonar; (b) side-scan sonar; (c) multibeam sonar.
图 2.典型声纳：（a）单波束声纳；（b）侧扫声纳；（c）多波束声纳。

Side-Scan Sonar: The side-scan sonar is composed of submodules such as the control unit, towed body, cable, and recorder, which aims at detailed study of topography, geology, and minerals and further performs object search and tracking. As shown in Fig. 2(b), the side-scan sonar emits directional pulse acoustic signals, where the horizontal beam angle is extremely small (less than 2 degrees) and the vertical beam angle is large (approximately 32 degrees). By analyzing the received acoustic image data, an object on the sea floor can be identified. However, side-scan sonar can only roughly estimate the direction of the object and cannot accurately measure the depth of the submarine.
侧扫声纳：侧扫声纳由控制单元、拖曳体、电缆和记录器 等子模块组成，旨在对地形、地质和矿物进行详细研究，并进一步执行目标搜索和跟踪。 如图 2（b）所示 ，侧扫声纳发射定向脉冲声波信号，其中水平波束角极小（小于 2 度），垂直波束角较大（约 32 度）。通过分析接收到的声学图像数据，可以识别海底的物体。不过，侧扫声纳只能大致估计物体的方向，无法准确测量潜艇的深度。

Multibeam Sonar: Multibeam sonar is the combination of multiple single-beam sonars, as shown in Fig. 2(c), which can obtain the high-precision direction and depth value of the submarine object by travel time. Compared with single-beam sonar, the multibeam system can provide larger coverage of the seabed area with faster speed and higher accuracy, which greatly improves the efficiency of ocean exploration.
多波束声纳：多波束声纳是由多个单波束声纳组合而成，如图 2（c）所示，它可以通过行进时间获得海底物体的高精度方向和深度值。与单波束声纳相比，多波束系统能以更快的速度和更高的精度覆盖更大的海底区域，大大提高了海洋探测的效率。

3.1.2. Underwater acoustic positioning sensors
3.1.2.水下声学定位传感器

Underwater acoustic positioning sensors can estimate the position of a measured object (e.g., underwater robot). Depending on the length of the baseline, underwater acoustic positioning systems can be divided into the following three categories: the ultrashort baseline (USBL), the short baseline (SBL), and the long baseline (LBL) positioning systems, as shown in Fig. 3.
水下声学定位传感器可以估算被测物体（如水下机器人） 的位置。根据基线长度的不同，水下声学定位系统可分为以下三类：超短基线（USBL）、短基线（SBL）和长基线（LBL）定位系统， 如图 3 所示 。

1. Download: Download high-res image (124KB)
   下载：下载高清图片 (124KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 3. Underwater acoustic positioning sensor: (a) ultrashort baseline; (b) short baseline; (c) long baseline.
图 3.水下声波定位传感器：（a）超短基线；（b）短基线；（c）长基线。

USBL and SBL: Both methods combine an array of acoustic transducers installed on a ship and a submarine transponder placed on an underwater robot. By providing the space transformation matrix between the array of acoustic transducers and the hull, we can estimate the relative pose by travel time and phase difference. The major difference between USBL and SBL is the space distances among the transceivers. For the USBL, the distance among transceivers is less than one meter. The SBL system generally contains more than three transducers with a distance of 20–50 m. By increasing the space distances, the measurement accuracy can be improved. However, the main drawback of these sensors is that they are difficult to calibrate.
USBL 和 SBL：这两种方法都是将安装在船上的声学传感器阵列和安装在水下机器人上的水下应答器结合起来。通过提供 声学传感器阵列与船体之间的空间 变换矩阵，我们可以通过移动时间和相位差来估计相对姿态。USBL 和 SBL 的主要区别在于收发器之间的空间距离。对于 USBL，收发器之间的距离不到一米。SBL 系统一般包含三个以上的收发器，距离为 20-50 米。不过，这些传感器的主要缺点是难以校准。

LBL: Different from the USBL, as shown in Figs. 3(b) and 3(c), LBL uses a set of acoustic transponders placed on the sea floor with a known relative position. Based on at least three acoustic beacons with a baseline length of 100 m to 20 km, we can compute the localization of robots within the coverage area of the acoustic signal. The LBL sensor can obtain high measurement accuracy and is not affected by the water depth, but the system is costly to establish and maintain because an underwater acoustic network needs to be deployed and recovered regularly.
LBL：与 USBL 不同，如图 3(b) 和 3(c) 所示，LBL 使用一组放置在海底的已知相对位置的声学应答器。根据至少三个基线长度为 100 米至 20 千米的声学信标，我们可以计算出声学信号覆盖范围内机器人的定位。LBL 传感器可以获得很高的测量精度，并且不受水深的影响，但该系统的建立和维护成本很高，因为需要定期部署和恢复水下声学网络。

3.2.1. Underwater Acoustics Image Detection/Tracking
3.2.1.水下声学图像探测/跟踪

Since sonar can acquire medium- and long-distance underwater acoustic image data, it is widely used in underwater object detection and tracking, as shown in Figs. 4(a) and 4(b). In the early period, object detection and tracking based on sonar were performed through expert processing (e.g., handcrafted selection). However, this method is time-consuming and affects the performance of sonar sensors. Then, researchers began to explore acoustic features (e.g., time, frequency, and time-frequency features). Among the above features, the time-frequency feature is suitable for nonstationary object detection and can obtain better results. For example, depending on the wavelet analysis and the relative Hilbert-Huang transform, Wang et al. [9] performed frequency decomposition and reconstruction of acoustic signals for underwater object detection.
由于声纳可以获取中远距离的水下声学图像数据，因此被广泛应用于水下物体探测和跟踪， 如图 4（a）和 4（b）所示 。早期，基于声纳的物体探测和跟踪是通过专家处理（如手工选择）来完成的。然而，这种方法耗时较长，而且会影响声纳传感器的性能。随后，研究人员开始探索声学特征（如时间、频率和时频特征）。在上述特征中，时频特征适用于非稳态物体检测，能获得更好的结果。例如，根据小波分析和相对希尔伯特-黄变换，Wang 等人 [9] 对声学信号进行了频率分解和重构，用于水下物体检测。

1. Download: Download high-res image (312KB)
   下载：下载高清图片 (312KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 4. Underwater acoustic object detection and scene reconstruction: (a) side-scan sonar object detection; (b) multibeam sonar object detection; (c) sparse reconstruction [10]; (d) dense reconstruction [11].
图 4.水下声学物体探测和场景重建：（a）侧扫声纳物体探测；（b）多波束声纳物体探测；（c）稀疏重建[10]；（d）密集重建[11]。

Recently, machine learning has achieved many progresses in object detection and classification. For example, Zhang et al. [10] summarized 19 conventional classifiers for underwater acoustic object detection (e.g., support vector machines (SVMs), K nearest neighbors (KNNs), and decision trees (DTs)). Ke et al. [11] proposed a supervised feature extraction algorithm to perform the object classification of underwater sonar images. In addition, deep learning is widely used in underwater acoustic signal analysis. Wang et al. [12] adopt deep neural networks (DNNs) to learn and fuse features and further utilize the Gaussian mixture model (GMM) to improve the performance. Phung et al. [13] proposed an unsupervised image statistics algorithm that combines deep semantic features to localize sonar targets.
最近，机器学习在物体检测和分类方面取得了许多进展。例如，Zhang等人[10]总结了 19 种用于水下声学物体检测的传统分类器（如 支持向量机(SVM)、K 近邻 (KNN) 和决策树 (DT)）。Ke 等人[11] 提出了一种监督特征提取算法，用于对水下声纳图像进行物体分类。此外，深度学习也被广泛应用于水下声学信号分析。Wang 等人[12] 采用深度神经网络（DNN）来学习和融合特征，并进一步利用 高斯混合模型（GMM）来提高性能。Phung 等人[13] 提出了一种结合深度语义特征的无监督图像统计算法来定位声纳目标。

3.2.2. Underwater Acoustics Image 3D Reconstruction
3.2.2.水下声学图像三维重建

Acoustic images also obtain significant developments in underwater acoustic 3D reconstruction (e.g., sparse reconstruction, dense reconstruction, and acoustic and optical fusion reconstruction), as shown in Figs. 4(c) and 4(d) [14], [15].
如图 4（c）和 4（d）所示，声学图像在水下声学三维重建（如稀疏重构、密集重构以及声光融合重构）方面也取得了重大进展[14]，[15 ]。

Sparse Reconstruction: Approaches to perform sparse reconstruction typically detect the corner points on the sonar image and further solve the elevation angle and relative attitude transformation based on the sampling method. However, these methods are susceptible to the degradation of the sonar sensor model and the initial value of the algorithm [16]. An alternative method is structure from motion, which tracks and matches the scene's sparse features and further performs sparse scene reconstruction. Huang et al. [17] designed an acoustic based structure from motion model named as ASFM, which adopts handcrafted features to recover the 3D location of landmarks. Subsequently, Yonghoon et al. [18] optimized the ASFM algorithm to improve feature extraction and data association modules for acoustic sparse reconstruction.
稀疏重建：进行稀疏重建的方法通常是检测声纳图像上的角点，并根据采样方法进一步解决仰角和相对姿态变换问题。然而，这些方法容易受到声纳传感器模型和算法初始值退化的影响 [16]。另一种方法是运动结构法，它可以跟踪和匹配场景的稀疏特征，并进一步进行稀疏场景重构。Huang等人[17]设计了一种基于声学结构的运动模型，命名为 ASFM，该模型采用手工特征来恢复地标的三维位置。随后，Yonghoon等人[18]优化了 ASFM 算法，改进了声学稀疏重建的特征提取和数据关联模块。

Dense Reconstruction: Acoustic dense reconstruction is a challenging problem because of the low signal-noise ratio of acoustic images. To address this problem, Zerr et al. [19] propose a two-step algorithm for dense reconstruction of the sea floor, which can generate a 3D target model by combining a height map and reflection map. Cho et al. [20] propose to estimate the pitch angle and assume that the top of the submarine target generates acoustic backscatter, which can obtain more details in the reconstructed results.
密集重建：声学密集重建是一个具有挑战性的问题，因为声学图像的信噪比很低。针对这一问题，Zerr等人[19]提出了一种两步海底密集重建算法，该算法可通过结合高度图和反射图生成三维目标模型。Cho等人[20] 提出估计俯仰角并假定潜艇目标顶部会产生声学反向散射，这样可以在重建结果中获得更多细节。

Acoustic and Optical Fusion Reconstruction: Methods based on acoustic and optical fusion reconstruction can combine the advantages of both sonar and visual images to improve underwater reconstruction performance. For example, Sharmin et al. [21] designed a navigation algorithm using sonar, vision and inertial information for underwater scene reconstruction. Since acoustic data provide robust information about underwater obstacles, the proposed method combines visual features with sonar features to optimize the robot position and 3D scene points. Subsequently, they also achieved high-precision 3D reconstruction results of underwater caves by combining sonar with vision sensors [22].
声光融合重建：基于声光融合重建的方法可以结合声纳和视觉图像的优势，提高水下重建性能。例如，Sharmin等人[21] 设计了一种利用声纳、视觉和惯性信息进行水下场景重建的导航算法。由于声学数据可提供有关水下障碍物的可靠信息，因此所提出的方法将视觉特征与声纳特征相结合，以优化机器人位置和三维场景点。随后，他们还通过将声纳与视觉传感器相结合，实现了水下洞穴的高精度三维重建结果 [22]。

4.1.1. Underwater Image Enhancement
4.1.1.水下图像增强

Due to the complex underwater scenario with light absorption and scattering, capturing a clear underwater images is still a challenging problem. Furthermore, these effects can reduce visibility and contrast [25]. As shown in Fig. 6(a), the image captured in the marine environment obtains low-quality results. By enhancing the image quality, we can achieve a high-quality image, as shown in Fig. 6(b). Generally, the methods of underwater image enhancement can be roughly subdivided into three categories (i.e., nonphysical model-based methods, physical model-based methods, and data-driven methods).
由于水下环境复杂，存在光吸收和散射现象，拍摄清晰的水下图像仍然是一个具有挑战性的问题。此外，这些影响会降低可见度和对比度 [25]。如图 6（a）所示，在海洋环境中拍摄的图像质量较低。通过增强图像质量，我们可以获得高质量的图像，如图 6(b) 所示。一般来说，水下图像增强方法可大致分为三类（即基于非物理模型的方法、基于物理模型的方法和数据驱动的方法）。

1. Download: Download high-res image (466KB)
   下载：下载高清图片 (466KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 6. Underwater 2D visual sensing: (a) low-quality underwater images; (b) underwater image quality restoration results [24]; (c) underwater object detection; (d) underwater object tracking.
图 6.水下二维视觉传感：（a）低质量水下图像；（b）水下图像质量恢复结果[24]；（c）水下物体检测；（d）水下物体跟踪。

Nonphysical model-based methods: These methods often enhance the image quality by changing the contrast and color space of the image. Iqbal et al. [26] adjusted the pixel range in HSV and RGB color space to enhance the underwater image quality. Based on the multiscale fusion strategy, Ancuti et al. [27] combined contrast enhancement with color correction methods to improve the image quality. However, since these methods do not build a real-world physical model for underwater image enhancement, they cannot obtain high-quality image results.
基于非物理模型的方法：这些方法通常通过改变图像的对比度和色彩空间来提高图像质量。Iqbal等人[26]调整了 HSV 和 RGB 色彩空间的像素范围，以提高水下图像质量。Ancuti等人[27]基于多尺度融合策略，将对比度增强与色彩校正方法相结合，提高了图像质量。然而，由于这些方法没有为水下图像增强建立真实世界的物理模型，因此无法获得高质量的图像结果。

Physical Model-Based Methods: Methods based on physical models improve image quality by building a physical model of degradation. (e.g., Jaffe-McGlamery equation), which can be denoted as:$I_c=J_c·e^{-{\beta }_c^D·z}+B_c^{\infty }·\left(1-e^{-{\beta }_c^B·z}\right)$where $I_c$ denotes the pixel value, $z$ is the distance from the object to the camera, $B_c^{\infty }$ denotes the veiling light, and $J_c$ is the true but unknown color of the 3D point. ${\beta }_c^D$ and ${\beta }_c^B$ represent the light attenuation coefficient.
基于物理模型的方法：基于物理模型的方法通过建立退化的物理模型来提高图像质量。(例如，Jaffe-McGlamery 方程），可表示为： $I_c=J_c·e^{-{\beta }_c^D·z}+B_c^{\infty }·\left(1-e^{-{\beta }_c^B·z}\right)$ ，其中 $I_c$ 表示像素值， $z$ 表示物体到摄像机的距离， $B_c^{\infty }$ 表示遮挡光线， $J_c$ 表示三维点真实但未知的颜色。 ${\beta }_c^D$ 和 ${\beta }_c^B$ 表示光衰减系数。

Based on the Jaffe-McGlamery equation, Galdran et al. [28], [29] considered that the absorption rate of red light is greater than both blue and green light and proposed a model based on the color channel of the image, which achieves image color restoration by establishing the relationship between the color value of the image and the medium parameters of the underwater environment. However, these methods require an accurate physical model of light degradation and a uniform distribution of the absorption coefficient. Therefore, these methods easily lead to unstable image restoration results. Recently, Akkaynak et al. [24] assumed that the absorption coefficient is not uniformly distributed in an underwater scenario and depends on the distance and reflection of the object. Based on this finding, they built an accurate underwater attenuation model to enhance the quality of the underwater image, which improves the robustness of image quality restoration.
Galdran等人[28]、[29] 在 Jaffe-McGlamery 方程的基础上，考虑到红光的吸收率大于蓝光和绿光，提出了基于图像颜色通道的模型，通过建立图像颜色值与水下环境介质参数之间的关系实现图像颜色还原。然而，这些方法需要精确的光降解物理模型和均匀分布的吸收系数。因此，这些方法很容易导致不稳定的图像复原结果。最近，Akkaynak等人[24]假设吸收系数在水下场景中并非均匀分布，而是取决于距离和物体的反射。基于这一发现，他们建立了一个精确的水下衰减模型来提高水下图像的质量，从而提高了图像质量恢复的鲁棒性。

Data-Driven Methods: These methods can be roughly subdivided into two categories (i.e., methods based on synthetic data and methods based on real data). More specifically, methods based on synthetic data are trained through artificially synthesized data pairs. However, underwater image models depend on special scenes, light conditions, temperature, and turbidity. Therefore, this method cannot be used conveniently in a practical environment. An alternative strategy is to train the model through real underwater data. Li et al. [30] propose an underwater image restoration model (i.e., WaterGAN), which combines the RGBD data in the air with the scene parameters to synthesize underwater images. Based on the synthetic data, they trained the deep learning model and enhanced the underwater image quality with a two-stage network. Li et al. [31] proposed a cycle-consistent weakly supervised underwater color restoration model, which trains the model with an adversarial network and can be used in an unknown water domain. However, due to the nature of multiple potential outputs [32], it tends to produce inauthentic results in some instances . Li et al. [33] built an underwater image enhancement dataset including low-quality underwater images and reference high-quality images and evaluated related underwater image enhancement algorithms. However, the performance of deep learning based methods are worse than physical model based models in terms of robustness and generalization.
数据驱动方法：这些方法大致可分为两类（即基于合成数据的方法和基于真实数据的方法）。更具体地说，基于合成数据的方法是通过人工合成的数据对进行训练。然而，水下图像模型取决于特殊的场景、光照条件、温度和浊度。因此，这种方法无法方便地应用于实际环境。另一种策略是通过真实的水下数据来训练模型。Li 等人[30]提出了一种水下图像复原模型（即 WaterGAN），该模型结合空气中的 RGBD 数据和场景参数合成水下图像。基于合成数据，他们训练了深度学习模型，并通过两级网络增强了水下图像质量。Li 等人[31]提出了一种循环一致的弱监督水下色彩还原模型，该模型采用对抗网络进行训练，可用于未知水域。然而，由于存在多个潜在输出[32]，在某些情况下往往会产生不真实的结果。Li等人[33] 建立了一个水下图像增强数据集，包括低质量水下图像和参考高质量图像，并评估了相关的水下图像增强算法。然而，基于深度学习的方法在鲁棒性和泛化方面的表现不如基于物理模型的模型。

4.1.2. Underwater Object Detection and Tracking
4.1.2.水下物体探测和跟踪

Underwater object detection aims to find the objects of interest in an image, as shown in Fig. 6(c). Generally, underwater object detection can be roughly divided into two categories (i.e., methods based on traditional features and methods based on deep learning).
水下物体检测的目的是在图像中找到感兴趣的物体，如图 6(c) 所示。一般来说，水下物体检测可大致分为两类（即基于传统特征的方法和基于深度学习的方法）。

Methods Based on Traditional Features: These methods use hand-crafted features (e.g., local binary patterns (LBP), histogram of oriented gradient (HOG), and Haar features (Haar)). The method includes the following three steps: 1) generating the multilevel image pyramid by down-sampling the original image in both the horizontal and vertical directions and 2) extracting the image features by sliding on the image pyramid with a fixed-size window and sending the extracted features to the extractor and classifier (e.g., SVM and AdaBoost).
基于传统特征的方法：这些方法使用手工制作的特征（如 局部二值模式（LBP）、定向梯度直方图（HOG）和哈氏特征（Haar））。该方法包括以下三个步骤：1) 通过在水平和垂直方向上对原始图像进行下采样，生成多级图像金字塔；2) 通过在图像金字塔上滑动固定大小的窗口提取图像特征，并将提取的特征发送给提取器和分类器（如 SVM和 AdaBoost）。

Methods Based on Deep Learning: These methods can be categorized into single-stage object detection methods and two-stage object detection methods. More specifically, the first method directly localizes objects by estimate the score of the bounding box directly that can be obtained by dense sampling on the input image (e.g., YOLO [34] and SSD [35]), which has fast detection speed but low accuracy. In contrast, the second method can obtain higher precision by generating object proposals and obtain more accurate results through region proposal networks (RPNs) (e.g., Faster RCNN [36]).
基于深度学习的方法：这些方法可分为单阶段物体检测方法和双阶段物体检测方法。具体来说，第一种方法通过对输入图像进行密集采样（如 YOLO[34]和 SSD[35 ]），直接估计边界框的得分来定位物体，这种方法检测速度快，但精度低。相比之下，第二种方法可以通过生成对象建议来获得更高的精度，并通过区域建议网络（RPN）获得更精确的结果（如 Faster RCNN[36]）。

Underwater object tracking involves continuously predicting and updating the statuses (scale, position, and rotation) of a specified object in the subsequent images, as shown in Fig. 6(d), which often encounters some problems (e.g., object deformation, object occlusion, similar object characteristics in the object and background, shadow and illumination changes). Classical methods can be divided into the following three categories: optical flow, mean shift, and convolutional network tracking (CNT). More specifically, the method based on the optical flow requires robust features in the image, which is not effective for motion blur; the method based on mean shift also requires features, which can track non-rigid objects efficiently and is robust to distance changes; CNTs can directly learn features from raw image data, which can extract the internal structure and local geometric information of the image and obtain better performance in this task.
水下物体跟踪是指连续预测和更新指定物体在后续图像中的状态（比例、位置和旋转），如图 6(d)所示，其中经常会遇到一些问题（如物体变形、物体遮挡、物体和背景中相似的物体特征、阴影和光照变化等）。经典方法可分为以下三类：光流、均值偏移和卷积网络跟踪（CNT）。具体来说，基于光流的方法需要图像中的稳健特征，对运动模糊效果不佳；基于均值平移的方法也需要特征，可以高效跟踪非刚性物体，对距离变化具有稳健性；卷积网络跟踪可以直接从原始图像数据中学习特征，从而提取图像的内部结构和局部几何信息，在这项任务中获得更好的性能。

4.2.1. Underwater Camera Calibration
4.2.1.水下摄像机校准

Underwater camera calibration that relates image pixels and 3D world points is a prerequisite for many tasks (e.g., 3D reconstruction and localization). Most studies ignore the influence of medium refraction and directly use the pinhole camera model in underwater scenarios. However, these methods often lead to inaccurate 3D reconstruction results [37]. To improve the accuracy of underwater camera calibration, many studies focus on exploring the impact of refraction on the underwater camera model. The Woods Hole Oceanographic Institute studies the ray-based underwater camera model and found that the back-projection rays do not intersect at one point, so the underwater camera model cannot be represented by a single-view pinhole camera model [37]. Agrawal et al. [38] proposed a refractive camera model, which can simplify the underwater camera model as an axis camera model. Furthermore, based on the axis camera model, the forward and backward projection equations can be well formulated. Chen et al. [39] proposed a three-wavelength dispersion method to calibrate an underwater camera. Tomasz et al. [40] built a precomputed lookup table to quickly correct refraction distortion, but they assumed that refraction distortion is not related with the depth of the scene and rectified all images through the same look-up table.
将图像像素与三维世界点联系起来的水下相机校准是许多任务（如三维重建和定位） 的先决条件。大多数研究都忽略了介质折射的影响，在水下场景中直接使用针孔摄像机模型。然而，这些方法往往会导致不准确的三维重建结果 [37]。为了提高水下摄像机校准的准确性，许多研究都侧重于探索折射对水下摄像机模型的影响。伍兹霍尔海洋研究所研究了基于射线的水下摄像机模型，发现背投射线并不相交于一点，因此水下摄像机模型不能用单视角针孔摄像机模型表示[37]。Agrawal等人[38] 提出了折射相机模型，可将水下相机模型简化为轴相机模型。此外，在轴相机模型的基础上，可以很好地建立前向和后向投影方程。Chen 等人[39]提出了校准水下摄像机的三波长色散方法。Tomasz等人[38]提出了一种校准水下摄像机的三波长色散方法。[40] 建立了一个预先计算的查找表来快速纠正折射失真，但他们假设折射失真与场景深度无关，并通过相同的查找表对所有图像进行纠正。

4.2.2. Underwater 3D Data Acquisition
4.2.2.水下三维数据采集

Some studies achieve 3D reconstruction using structured light, laser imaging (e.g., TOF), and multiview reconstruction. Among the above techniques, the structured light with high precision has been used for underwater 3D data acquisition. Therefore, we mainly introduce underwater structured light 3D reconstruction in this subsection.
一些研究利用结构光、激光成像（如 TOF）和多视角重建实现了三维重建。在上述技术中，高精度的结构光已被用于水下三维数据采集。因此，本小节主要介绍水下结构光三维重建。

Underwater structured light sensors acquire the 3D point cloud by computing the intersection point between the laser light plane and the camera beam. Bodenmann et al. [41] designed an underwater laser scanner, as shown in Fig. 7(a), which can simultaneously capture the gray image and laser stripe image. Furthermore, they transform the collected color image data into 3D point clouds, which can obtain accurate shape and color information. Bleier et al. [42] designed a cross-line laser scanner, as shown in Fig. 7(b), where the camera captures the cross-line projected by the laser and computes the 3D point clouds by ray-plane triangulation. However, these methods ignore the influence of medium refraction in underwater environments. Palomer et al [43] developed an underwater scene scanning sensor with only laser rotation, which uses a stepper motor to quickly rotate the laser in the field of view and can perform 3D scene reconstruction in the field of view in a short time. Subsequently, Palomer et al [43], [44] found that the laser plane entering the underwater scenario is refracted twice and cannot be described by a plane equation. Therefore, they denote the light plane as an elliptic cone and obtain higher 3D reconstruction accuracy with ray-cone triangulation. They also install the designed laser scanner on the underwater robot to perform underwater robot grasping, as shown in Fig. 7(c). Recently, the Shenyang Institute of Automation designed an underwater structured light sensor, as shown in Fig. 7(d). By explicitly considering medium refraction, the sensor can achieve high-precision reconstruction performance [45].
水下结构光传感器通过计算激光光平面与相机光束之间的交点来获取三维点云。Bodenmann等人[41] 设计了一种水下激光扫描仪 ，如图 7(a) 所示 ，可以同时采集灰度图像和激光条纹图像。此外，他们还将采集到的彩色图像数据转化为三维点云，从而获得精确的形状和颜色信息。Bleier等人[42]设计了一种交叉线激光扫描仪，如图 7(b) 所示，相机捕捉激光投射的交叉线，并通过射线平面三角测量法计算三维点云。然而，这些方法都忽略了水下环境中介质折射的影响。Palomer等人[43] 开发了一种仅靠激光旋转的水下场景扫描传感器，它利用步进电机在 视场 中快速旋转激光 ，可在短时间内完成视场中的三维场景重建。 随后，Palomer等人[43]、[44] 发现，进入水下场景的激光平面会发生两次折射，无法用平面方程描述。因此，他们将光面表示为椭圆锥，并通过射线锥三角测量法获得了更高的三维重建精度。他们还将设计的激光扫描仪安装在水下机器人上，实现了水下机器人抓取，如图 7（c）所示。最近，沈阳自动化研究所设计了一种水下结构光传感器，如图 7(d) 所示。通过明确考虑介质折射，该传感器可实现高精度重建性能[45]。

1. Download: Download high-res image (385KB)
   下载：下载高清图片 (385KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 7. Underwater laser scanner sensor: (a) underwater structured light measurement sensor designed by University of Tokyo [41]; (b) underwater cross-laser scanning sensor designed by Julius-Maximilians-University Wurzburg [42]; (c) underwater rotation laser scanner designed by University of Girona [43]; (d) underwater laser scanner sensor designed by Shenyang Institute of Automation, Chinese Academy of Sciences [45].
图 7. 水下激光扫描传感器水下激光扫描传感器：（a）东京大学设计的水下结构光测量传感器 [41]；（b）沃茨堡朱利叶斯-马克西米利安大学设计的水下交叉激光扫描传感器[ 42 ]；（c）赫罗纳大学设计的水下旋转激光扫描仪[43]；（d）中国科学院沈阳自动化研究所设计的水下激光扫描传感器[45]。

4.2.3. Underwater Multi-View 3D Reconstruction
4.2.3.水下多视角三维重建

Underwater multiview 3D reconstruction can simultaneously estimate the camera pose and reconstruct the 3D scene structure based on the 2D re-projection error, where the minimization function can be denoted as:$T_{k,k-1}=\underset{T_{kk-1}}{\text{argmin}}\sum _{i=1}^N\parallel q_k^i-\pi \left(T_{k,k-1},Q_{K-1}^i\right){\parallel }^2$where $q_k^i$ is the captured $i^{th}$ feature that occurs in the $k^{th}$ image. $T_{k,k-1}$ is the spatial transformation matrix from the $k-1^{th}$ image to the $k^{th}$ image. $Q_{K-1}^i$ denotes the 3D point. $\pi \left(T_{k,k-1},Q_{K-1}^i\right)$ denotes the projection function. Classical multiview 3D reconstruction can be classified into the following two classes: simultaneous localization and mapping (SLAM) [[46] and structure from motion (SFM) [47]].
水下多视角三维重建可同时估计摄像机姿态并基于二维重投影误差重建三维场景结构，其中最小化函数可表示为： $T_{k,k-1}=\underset{T_{kk-1}}{\text{argmin}}\sum _{i=1}^N\parallel q_k^i-\pi \left(T_{k,k-1},Q_{K-1}^i\right){\parallel }^2$ ，其中 $q_k^i$ 是 在 $k^{th}$ 图像中出现的捕捉到的 $i^{th}$ 特征。 $T_{k,k-1}$ 是 $k-1^{th}$ 图像到 $k^{th}$ 图像的 空间变换矩阵 。 $Q_{K-1}^i$ 表示三维点。 $\pi \left(T_{k,k-1},Q_{K-1}^i\right)$ 表示投影函数。经典的多视角三维重建可分为以下两类：同步定位与映射（SLAM）[[46]和运动结构（SFM）[47]]。

Underwater SLAM: SLAM can estimate the robot's pose and reconstruct the 3D scene structure simultaneously. Due to the complicated underwater environment (e.g., scattering, absorption, turbidity, and refraction), underwater SLAM is a challenging problem [48]. Ferrera et al [49] collected multiple underwater datasets for the underwater SLAM task, where the trajectories computed by the structure-from-motion (SfM) library Colmap were used as reference trajectories. Bharat et al. [46] also collected an underwater SLAM dataset with an underwater robot, where they performed underwater trajectory estimation and evaluated the performance of the most recent open-source packages (e.g., LSD-SLAM, DSO, SVO, ORB-SLAM2, ROVIO, OKVIS, VINS-Mono) on the collected datasets, as shown in Fig. 8(a). Table 1 summarizes the characteristics of the open-source SLAM method. From the evaluated results, OKVIS and ROVIO can achieve good results in terms of robustness and accuracy; ORB-SLAM2, SVO, and VINS-Mono achieve good results in terms of the absolute scale; based on minimizing a photometric error, DSO can obtain dense 3D reconstruction results.
水下 SLAM：SLAM 可以同时估计机器人的姿态和重建三维场景结构。由于水下环境复杂（如散射、吸收、浊度和折射），水下 SLAM 是一个具有挑战性的问题[48]。Ferrera等人[49] 为水下 SLAM 任务收集了多个水下数据集，并将结构运动（SfM）库 Colmap 计算出的轨迹作为参考轨迹。Bharat 等人[46]也收集了水下机器人的水下 SLAM 数据集，他们在数据集上进行了水下轨迹估计，并评估了最新开源软件包（如 LSD-SLAM、DSO、SVO、ORB-SLAM2、ROVIO、OKVIS、VINS-Mono）的性能，如图 8(a)所示。表 1总结了开源 SLAM 方法的特点。从评估结果来看，OKVIS 和 ROVIO 在鲁棒性和准确性方面都能达到很好的效果；ORB-SLAM2、SVO 和 VINS-Mono 在绝对尺度方面都能达到很好的效果；DSO 在最小化光度误差的基础上，能获得高密度的三维重建结果。

1. Download: Download high-res image (361KB)
   下载：下载高清图片 (361KB)
2. Download: Download full-size image
   下载：下载全尺寸图片

Fig. 8. Underwater multiview 3D reconstruction: (a) underwater SLAM [46]; (b) underwater SFM [49].
图 8.水下多视角三维重建：（a）水下 SLAM[46]；（b）水下 SFM [49]。

Table 1. Compare the SLAM methods [46].
表 1.SLAM 方法比较[46]。

Method 方法	Camera 照相机	IMU	Loop Closure 环形闭合
LSD-SLAM	mono 单声道	×	√
DSO	mono 单声道	×	×
SVO	multi 多	Optional 可选	×
ORB-SLAM2	Mono, stereo 单声道、立体声	×	√
ROVIO	multi 多	√	×
OKVIS	multi 多	√	×
VINS-Mono	mono 单声道	√	√

Underwater SFM: SFM recovers the 3D scene structure from unordered images. Most existing methods ignore the influence of medium variation and adopt the pinhole camera model for underwater SFM directly, as shown in Fig. 8(b) [50]. However, if we adopt the pinhole camera model for underwater SFM, the systematic geometric bias will interfere with the precision of the 3D reconstruction because of the multirefraction between different media. Recently, some methods have begun to consider the influence of medium refraction for 3D reconstruction in underwater scenarios. To perform underwater refractive pose estimation, Jordt et al. [51] formulate a cost function by combining a real camera (refractive camera model) and a virtual camera (pinhole camera model). Along with refractive pose estimation, they perform underwater 3D reconstruction based on minimizing the 2D reprojection error. Chadebecq et al. [52] introduced Plucker coordinates and constructed a refractive fundamental matrix for underwater object and fluid-immersed organ reconstruction. However, these methods of underwater SFM often require a complex computation process and cannot perform underwater SFM efficiently.
水下 SFM：SFM 可从无序图像中恢复三维场景结构。现有的大多数方法都忽略了介质变化的影响，直接采用针孔摄像机模型进行水下 SFM， 如图 8(b)所示 [50]。然而，如果采用针孔摄像机模型进行水下 SFM，由于不同介质之间存在多重折射，系统性几何偏差会干扰三维重建的精度。最近，一些方法开始考虑介质折射对水下场景三维重建的影响。为了进行水下折射姿态估计，Jordt等人[51] 结合真实摄像机（折射摄像机模型）和虚拟摄像机（针孔摄像机模型）制定了一个成本函数。在进行折射姿态估计的同时，他们还根据二维重投影误差最小化原则进行水下三维重建。Chadebecq 等人[52] 引入了 Plucker 坐标，并构建了用于水下物体和流体浸没器官重建的折射基本矩阵。然而，这些水下 SFM 方法往往需要复杂的计算过程，无法高效地进行水下 SFM。