Code release for the paper PointRCNN:3D Object Proposal Generation and Detection from Point Cloud, CVPR 2019.
论文PointRCNN的代码发布：从点云生成3D对象提案和检测，CVPR 2019。

Authors: Shaoshuai Shi, Xiaogang Wang, Hongsheng Li.
作者：石绍帅，王晓刚，李洪生。

[arXiv]  [Project Page] 
[arXiv][项目页面]

New: We have provided another implementation of PointRCNN for joint training with multi-class in a general 3D object detection toolbox [OpenPCDet].
新：我们提供了PointRCNN的另一种实现，用于在通用3D对象检测工具箱[OpenPCDet]中进行多类联合训练。

In this work, we propose the PointRCNN 3D object detector to directly generated accurate 3D box proposals from raw point cloud in a bottom-up manner, which are then refined in the canonical coordinate by the proposed bin-based 3D box regression loss. To the best of our knowledge, PointRCNN is the first two-stage 3D object detector for 3D object detection by using only the raw point cloud as input. PointRCNN is evaluated on the KITTI dataset and achieves state-of-the-art performance on the KITTI 3D object detection leaderboard among all published works at the time of submission.
在这项工作中，我们提出了PointRCNN 3D对象检测器，以自下而上的方式从原始点云直接生成准确的3D框建议，然后通过提出的基于bin的3D框回归损失在规范坐标中进行细化。据我们所知，PointRCNN是第一个只使用原始点云作为输入的两阶段3D对象检测器。PointRCNN在KITTI数据集上进行了评估，并在提交时的所有已发表作品中在KITTI 3D对象检测排行榜上实现了最先进的性能。

For more details of PointRCNN, please refer to our paper or project page.
有关PointRCNN的更多详细信息，请参阅我们的论文或项目页面。

• Multiple GPUs for training
  用于训练的多个GPU
• GPU version rotated NMS  GPU版本旋转NMS
• Faster PointNet++ inference and training supported by Pointnet2.PyTorch
  Pointnet2.PyTorch支持更快的PointNet++推理和训练
• PyTorch 1.0
• TensorboardX
• Still in progress  仍在进行

All the codes are tested in the following environment:
所有代码均在以下环境中进行测试：

• Linux (tested on Ubuntu 14.04/16.04)
  Linux（在Ubuntu 14.04/16.04上测试）
• Python 3.6+
• PyTorch 1.0

a. Clone the PointRCNN repository.
a.克隆PointRCNN存储库。

git clone --recursive https://github.com/sshaoshuai/PointRCNN.git

If you forget to add the --recursive parameter, just run the following command to clone the Pointnet2.PyTorch submodule.
如果忘记添加--recursive参数，只需运行以下命令来克隆Pointnet2.PyTorch子模块。

git submodule update --init --recursive

b. Install the dependent python libraries like easydict,tqdm, tensorboardX etc.
B.安装依赖的python库，如easydict，tqdm，tensorboardX等。

c. Build and install the pointnet2_lib, iou3d, roipool3d libraries by executing the following command:
C.通过执行以下命令构建并安装pointnet2_lib、iou 3d、roipool 3d库：

sh build_and_install.sh

Please download the official KITTI 3D object detection dataset and organize the downloaded files as follows:
请下载官方KITTI 3D物体检测数据集，并按以下方式组织下载的文件：

PointRCNN
├── data
│   ├── KITTI
│   │   ├── ImageSets
│   │   ├── object
│   │   │   ├──training
│   │   │      ├──calib & velodyne & label_2 & image_2 & (optional: planes)
│   │   │   ├──testing
│   │   │      ├──calib & velodyne & image_2
├── lib
├── pointnet2_lib
├── tools

Here the images are only used for visualization and the road planes are optional for data augmentation in the training.
在这里，图像仅用于可视化，道路平面可用于训练中的数据增强。

You could download the pretrained model(Car) of PointRCNN from here(~15MB), which is trained on the train split (3712 samples) and evaluated on the val split (3769 samples) and test split (7518 samples). The performance on validation set is as follows:
您可以从这里下载PointRCNN的预训练模型（Car）（约15 MB），它在训练分割（3712个样本）上进行训练，并在瓦尔分割（3769个样本）和测试分割（7518个样本）上进行评估。验证集的性能如下：

Car AP@0.70, 0.70, 0.70:
bbox AP:96.91, 89.53, 88.74
bev  AP:90.21, 87.89, 85.51
3d   AP:89.19, 78.85, 77.91
aos  AP:96.90, 89.41, 88.54

You could run the following command to evaluate the pretrained model (set RPN.LOC_XZ_FINE=False since it is a little different with the default configuration):
您可以运行以下命令来评估预训练模型（设置RPN. XZ_FINE=False，因为它与默认配置略有不同）：

python eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt PointRCNN.pth --batch_size 1 --eval_mode rcnn --set RPN.LOC_XZ_FINE False

• To evaluate a single checkpoint, run the following command with --ckpt to specify the checkpoint to be evaluated:
  要评估单个检查点，请使用--ckpt运行以下命令以指定要评估的检查点：

python eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt ../output/rpn/ckpt/checkpoint_epoch_200.pth --batch_size 4 --eval_mode rcnn 

• To evaluate all the checkpoints of a specific training config file, add the --eval_all argument, and run the command as follows:
  要评估特定训练配置文件的所有检查点，请添加--eval_all参数，然后运行命令，如下所示：

python eval_rcnn.py --cfg_file cfgs/default.yaml --eval_mode rcnn --eval_all

• To generate the results on the test split, please modify the TEST.SPLIT=TEST and add the --test argument.
  要生成测试拆分的结果，请修改TEST.SPLIT=TEST并添加--test参数。

Here you could specify a bigger --batch_size for faster inference based on your GPU memory. Note that the --eval_mode argument should be consistent with the --train_mode used in the training process. If you are using --eval_mode=rcnn_offline, then you should use --rcnn_eval_roi_dir and --rcnn_eval_feature_dir to specify the saved features and proposals of the validation set. Please refer to the training section for more details.
在这里，您可以指定一个更大的--batch_size，以便根据GPU内存进行更快的推理。注意--eval_mode参数应该与训练过程中使用的--train_mode一致。如果使用--eval_mode=rcnn_offline，则应使用--rcnn_eval_roi_dir和--rcnn_eval_feature_dir指定验证集的已保存功能和建议。请参阅培训部分了解更多详情。

Currently, the two stages of PointRCNN are trained separately. Firstly, to use the ground truth sampling data augmentation for training, we should generate the ground truth database as follows:
目前，PointRCNN的两个阶段是分开训练的。首先，为了使用地面实况采样数据增强进行训练，我们应该如下生成地面实况数据库：

python generate_gt_database.py --class_name 'Car' --split train

• To train the first proposal generation stage of PointRCNN with a single GPU, run the following command:
  要使用单个GPU训练PointRCNN的第一个提案生成阶段，请运行以下命令：

python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 16 --train_mode rpn --epochs 200

• To use mutiple GPUs for training, simply add the --mgpus argument as follows:
  要使用多个GPU进行训练，只需添加--mgpus参数，如下所示：

CUDA_VISIBLE_DEVICES=0,1 python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 16 --train_mode rpn --epochs 200 --mgpus

After training, the checkpoints and training logs will be saved to the corresponding directory according to the name of your configuration file. Such as for the default.yaml, you could find the checkpoints and logs in the following directory:
训练结束后，检查点和训练日志将根据您的配置文件名保存到相应的目录中。例如对于default.yaml，您可以在以下目录中找到检查点和日志：

PointRCNN/output/rpn/default/

which will be used for the training of RCNN stage.
其将用于RCNN阶段的训练。

Suppose you have a well-trained RPN model saved at output/rpn/default/ckpt/checkpoint_epoch_200.pth, then there are two strategies to train the second stage of PointRCNN.
假设您在 output/rpn/default/ckpt/checkpoint_epoch_200.pth 保存了一个经过良好训练的RPN模型，那么有两种策略来训练PointRCNN的第二阶段。

(a) Train RCNN network with fixed RPN network to use online GT augmentation: Use --rpn_ckpt to specify the path of a well-trained RPN model and run the command as follows:
(a)使用固定RPN网络训练RCNN网络以使用在线GT增强：使用--rpn_ckpt指定经过良好训练的RPN模型的路径，并运行命令如下：

python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --train_mode rcnn --epochs 70  --ckpt_save_interval 2 --rpn_ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth

(b) Train RCNN network with offline GT augmentation:
(b)用离线GT增强训练RCNN网络：

1. Generate the augmented offline scenes by running the following command:
   通过运行以下命令生成增强的离线场景：

python generate_aug_scene.py --class_name Car --split train --aug_times 4

2. Save the RPN features and proposals by adding --save_rpn_feature:
   通过添加--save_rpn_功能：保存RPN功能和建议

• To save features and proposals for the training, we set TEST.RPN_POST_NMS_TOP_N=300 and TEST.RPN_NMS_THRESH=0.85 as follows:
  为了保存训练的功能和建议，我们设置TEST.RPN_POST_NMS_TOP_N=300和TEST.RPN_NMS_THRESH=0.85，如下所示：

python eval_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --eval_mode rpn --ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth --save_rpn_feature --set TEST.SPLIT train_aug TEST.RPN_POST_NMS_TOP_N 300 TEST.RPN_NMS_THRESH 0.85

• To save features and proposals for the evaluation, we keep TEST.RPN_POST_NMS_TOP_N=100 and TEST.RPN_NMS_THRESH=0.8 as default:
  为了保存评估的功能和建议，我们将TEST.RPN_POST_NMS_TOP_N=100和TEST.RPN_NMS_THRESH=0.8作为默认值：

python eval_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --eval_mode rpn --ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth --save_rpn_feature

3. Now we could train our RCNN network. Note that you should modify TRAIN.SPLIT=train_aug to use the augmented scenes for the training, and use --rcnn_training_roi_dir and --rcnn_training_feature_dir to specify the saved features and proposals in the above step:
   现在我们可以训练我们的RCNN网络。请注意，您应该修改TRAIN.SPLIT=train_aug以使用增强场景进行训练，并使用--rcnn_training_roi_dir和--rcnn_training_feature_dir指定上述步骤中保存的功能和建议：

python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --train_mode rcnn_offline --epochs 30  --ckpt_save_interval 1 --rcnn_training_roi_dir ../output/rpn/default/eval/epoch_200/train_aug/detections/data --rcnn_training_feature_dir ../output/rpn/default/eval/epoch_200/train_aug/features

For the offline GT sampling augmentation, the default setting to train the RCNN network is RCNN.ROI_SAMPLE_JIT=True, which means that we sample the RoIs and calculate their GTs in the GPU. I also provide the CPU version proposal sampling, which is implemented in the dataloader, and you could enable this feature by setting RCNN.ROI_SAMPLE_JIT=False. Typically the CPU version is faster but costs more CPU resources since they use mutiple workers.
对于离线GT采样增强，训练RCNN网络的默认设置是RCNN.ROI_SAMPLE_JIT=True，这意味着我们对ROI进行采样并在GPU中计算其GT。我还提供了CPU版本建议采样，它在数据加载器中实现，您可以通过设置RCNN.ROI_SAMPLE_JIT=False来启用此功能。通常，CPU版本更快，但由于使用多个工作线程，因此会消耗更多的CPU资源。

All the codes supported mutiple GPUs, simply add the --mgpus argument as above. And you could also increase the --batch_size by using multiple GPUs for training.
所有代码都支持多个GPU，只需添加--mgpus参数即可。你还可以通过使用多个GPU进行训练来增加--batch_size。

Note:
注意事项：

• The strategy (a), online augmentation, is more elegant and easy to train.
  策略（a），在线增强，更优雅，更容易训练。
• The best model is trained by the offline augmentation strategy with CPU proposal sampling (set RCNN.ROI_SAMPLE_JIT=False).
  最好的模型通过离线增强策略和CPU建议采样（设置RCNN.ROI_SAMPLE_JIT=False）进行训练。
• Theoretically, the online augmentation should be better, but currently the online augmentation is a bit lower than the offline augmentation, and I still didn't know why. All discussions are welcomed.
  从理论上讲，在线增强应该更好，但目前在线增强比离线增强低一点，我仍然不知道为什么。欢迎所有讨论。
• I am still working on this codes to make it more stable.
  我还在努力使这个代码更稳定。

If you find this work useful in your research, please consider cite:
如果你发现这项工作对你的研究有用，请考虑引用：

@InProceedings{Shi_2019_CVPR,
    author = {Shi, Shaoshuai and Wang, Xiaogang and Li, Hongsheng},
    title = {PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}