[Enhance] Refactor doc

data/scannet/README.md

@@ -28,8 +28,11 @@ scannet
 ├── scans
 ├── scannet_train_instance_data
 ├── points
+│ ├── xxxxx.bin
 ├── instance_mask
+│ ├── xxxxx.bin
 ├── semantic_mask
+│ ├── xxxxx.bin
 ├── scannet_infos_train.pkl
 ├── scannet_infos_val.pkl
 

docs/1_exist_data_model.md

@@ -0,0 +1,222 @@
+# 1: Inference and train with existing models and standard datasets
+
+## Inference with existing models
+
+Here we provide testing scripts to evaluate a whole dataset (SUNRGBD, ScanNet, KITTI, etc.).
+
+For high-level apis easier to integrated into other projects and basic demos, please refer to Verification/Demo under [Get Started](./getting_started.md).
+
+### Test existing models on standard datasets
+
+- single GPU
+- single node multiple GPU
+- multiple node
+
+You can use the following commands to test a dataset.
+
+```shell
+# single-gpu testing
+python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] [--show]
+
+# multi-gpu testing
+./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
+```
+
+Optional arguments:
+- `RESULT_FILE`: Filename of the output results in pickle format. If not specified, the results will not be saved to a file.
+- `EVAL_METRICS`: Items to be evaluated on the results. Allowed values depend on the dataset, e.g., `proposal_fast`, `proposal`, `bbox`, `segm` are available for COCO, `mAP`, `recall` for PASCAL VOC. Cityscapes could be evaluated by `cityscapes` as well as all COCO metrics.
+- `--show`: If specified, detection results will be plotted in the silient mode. It is only applicable to single GPU testing and used for debugging and visualization. This should be used with `--show-dir`.
+- `--show-dir`: If specified, detection results will be plotted on the `***_points.obj` and `***_pred.ply` files in the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.
+
+Examples:
+
+Assume that you have already downloaded the checkpoints to the directory `checkpoints/`.
+
+1. Test votenet on ScanNet and save the points and prediction visualization results.
+
+ ```shell
+ python tools/test.py configs/votenet/votenet_8x8_scannet-3d-18class.py \
+ checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
+ --show --show-dir ./data/scannet/show_results
+ ```
+
+2. Test votenet on ScanNet, save the points, prediction, groundtruth visualization results, and evaluate the mAP.
+
+ ```shell
+ python tools/test.py configs/votenet/votenet_8x8_scannet-3d-18class.py \
+ checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
+ --eval mAP
+ --options 'show=True' 'out_dir=./data/scannet/show_results'
+ ```
+
+3. Test votenet on ScanNet (without saving the test results) and evaluate the mAP.
+
+ ```shell
+ python tools/test.py configs/votenet/votenet_8x8_scannet-3d-18class.py \
+ checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
+ --eval mAP
+ ```
+
+4. Test SECOND with 8 GPUs, and evaluate the mAP.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/hv_second_secfpn_6x8_80e_kitti-3d-3class.py \
+ checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
+ --out results.pkl --eval mAP
+ ```
+
+5. Test PointPillars on nuscenes with 8 GPUs, and generate the json file to be submit to the official evaluation server.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d.py \
+ checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth \
+ --format-only --options 'jsonfile_prefix=./pointpillars_nuscenes_results'
+ ```
+
+ The generated results be under `./pointpillars_nuscenes_results` directory.
+
+6. Test SECOND on KITTI with 8 GPUs, and generate the pkl files and submission datas to be submit to the official evaluation server.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/hv_second_secfpn_6x8_80e_kitti-3d-3class.py \
+ checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
+ --format-only --options 'pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results'
+ ```
+
+ The generated results be under `./second_kitti_results` directory.
+
+7. Test PointPillars on Lyft with 8 GPUs, generate the pkl files and make a submission to the leaderboard.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d.py \
+ checkpoints/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d_latest.pth --out results/pp_lyft/results_challenge.pkl \
+ --format-only --options 'jsonfile_prefix=results/pp_lyft/results_challenge' \
+ 'csv_path=results/pp_lyft/results_challenge.csv'
+ ```
+
+ **Notice**: To generate submissions on Lyft, `csv_path` must be given in the options. After generating the csv file, you can make a submission with kaggle commands given on the [website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit).
+
+7. Test PointPillars on waymo with 8 GPUs, and evaluate the mAP with waymo metrics.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
+ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
+ --eval waymo --options 'pklfile_prefix=results/waymo-car/kitti_results' \
+ 'submission_prefix=results/waymo-car/kitti_results'
+ ```
+
+ **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the options for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evalution, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision.
+
+8. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.
+
+ ```shell
+ ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
+ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
+ --format-only --options 'pklfile_prefix=results/waymo-car/kitti_results' \
+ 'submission_prefix=results/waymo-car/kitti_results'
+ ```
+
+ **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission.
+
+## Train predefined models on standard datasets
+
+MMDetection implements distributed training and non-distributed training,
+which uses `MMDistributedDataParallel` and `MMDataParallel` respectively.
+
+All outputs (log files and checkpoints) will be saved to the working directory,
+which is specified by `work_dir` in the config file.
+
+By default we evaluate the model on the validation set after each epoch, you can change the evaluation interval by adding the interval argument in the training config.
+```python
+evaluation = dict(interval=12) # This evaluate the model per 12 epoch.
+```
+
+**Important**: The default learning rate in config files is for 8 GPUs and the exact batch size is marked by the config's file name, e.g. '2x8' means 2 samples per GPU using 8 GPUs.
+According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU, e.g., lr=0.01 for 4 GPUs * 2 img/gpu and lr=0.08 for 16 GPUs * 4 img/gpu. However, since most of the models in this repo use ADAM rather than SGD for optimization, the rule may not hold and users need to tune the learning rate by themselves.
+
+### Train with a single GPU
+
+```shell
+python tools/train.py ${CONFIG_FILE} [optional arguments]
+```
+
+If you want to specify the working directory in the command, you can add an argument `--work_dir ${YOUR_WORK_DIR}`.
+
+### Train with multiple GPUs
+
+```shell
+./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
+```
+
+Optional arguments are:
+
+- `--no-validate` (**not suggested**): By default, the codebase will perform evaluation at every k (default value is 1, which can be modified like [this](https://github.com/open-mmlab/mmdetection/blob/master/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py#L174)) epochs during the training. To disable this behavior, use `--no-validate`.
+- `--work-dir ${WORK_DIR}`: Override the working directory specified in the config file.
+- `--resume-from ${CHECKPOINT_FILE}`: Resume from a previous checkpoint file.
+- `--options 'Key=value'`: Overide some settings in the used config.
+
+Difference between `resume-from` and `load-from`:
+`resume-from` loads both the model weights and optimizer status, and the epoch is also inherited from the specified checkpoint. It is usually used for resuming the training process that is interrupted accidentally.
+`load-from` only loads the model weights and the training epoch starts from 0. It is usually used for finetuning.
+
+### Train with multiple machines
+
+If you run MMDetection on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`. (This script also supports single machine training.)
+
+```shell
+[GPUS=${GPUS}] ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR}
+```
+
+Here is an example of using 16 GPUs to train Mask R-CNN on the dev partition.
+
+```shell
+GPUS=16 ./tools/slurm_train.sh dev mask_r50_1x configs/mask_rcnn_r50_fpn_1x_coco.py /nfs/xxxx/mask_rcnn_r50_fpn_1x
+```
+
+You can check [slurm_train.sh](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) for full arguments and environment variables.
+
+If you have just multiple machines connected with ethernet, you can refer to
+PyTorch [launch utility](https://pytorch.org/docs/stable/distributed_deprecated.html#launch-utility).
+Usually it is slow if you do not have high speed networking like InfiniBand.
+
+### Launch multiple jobs on a single machine
+
+If you launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,
+you need to specify different ports (29500 by default) for each job to avoid communication conflict.
+
+If you use `dist_train.sh` to launch training jobs, you can set the port in commands.
+
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_train.sh ${CONFIG_FILE} 4
+CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh ${CONFIG_FILE} 4
+```
+
+If you use launch training jobs with Slurm, there are two ways to specify the ports.
+
+1. Set the port through `--options`. This is more recommended since it does not change the original configs.
+
+ ```shell
+ CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --options 'dist_params.port=29500'
+ CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --options 'dist_params.port=29501'
+ ```
+
+2. Modify the config files (usually the 6th line from the bottom in config files) to set different communication ports.
+
+ In `config1.py`,
+
+ ```python
+ dist_params = dict(backend='nccl', port=29500)
+ ```
+
+ In `config2.py`,
+
+ ```python
+ dist_params = dict(backend='nccl', port=29501)
+ ```
+
+ Then you can launch two jobs with `config1.py` ang `config2.py`.
+
+ ```shell
+ CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
+ CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
+ ```

docs/2_new_data_model.md

@@ -0,0 +1,104 @@
+# 2: Train with customized datasets
+
+In this note, you will know how to train and test predefined models with customized datasets. We use the Waymo dataset as an example to describe the whole process.
+
+The basic steps are as below:
+
+1. Prepare the customized dataset
+2. Prepare a config
+3. Train, test, inference models on the customized dataset.
+
+## Prepare the customized dataset
+
+There are three ways to support a new dataset in MMDetection3D:
+
+1. reorganize the dataset into existing format.
+2. reorganize the dataset into a middle format.
+3. implement a new dataset.
+
+Usually we recommend to use the first two methods which are usually easier than the third.
+
+In this note, we give an example for converting the data into KITTI format.
+
+**Note**: We take Waymo as the example here considering its format is totally different from other existing formats. For other datasets using similar methods to organize data, like Lyft compared to nuScenes, it would be easier to directly implement the new data converter (for the second approach above) instead of converting it to another format (for the first approach above).
+
+### KITTI dataset format
+
+Firstly, the raw data for 3D object detection from KITTI are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation/testing set, `calib` contains calibration information files, `image_2` and `velodyne` include image data and point cloud data, and `label_2` includes label files for 3D detection.
+
+```
+mmdetection3d
+├── mmdet3d
+├── tools
+├── configs
+├── data
+│ ├── kitti
+│ │ ├── ImageSets
+│ │ ├── testing
+│ │ │ ├── calib
+│ │ │ ├── image_2
+│ │ │ ├── velodyne
+│ │ ├── training
+│ │ │ ├── calib
+│ │ │ ├── image_2
+│ │ │ ├── label_2
+│ │ │ ├── velodyne
+```
+
+Specific annotation format is described in the official object development [kit](https://s3.eu-central-1.amazonaws.com/avg-kitti/devkit_object.zip). For example, it consists of the following labels:
+
+```
+#Values Name Description
+----------------------------------------------------------------------------
+ 1 type Describes the type of object: 'Car', 'Van', 'Truck',
+ 'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram',
+ 'Misc' or 'DontCare'
+ 1 truncated Float from 0 (non-truncated) to 1 (truncated), where
+ truncated refers to the object leaving image boundaries
+ 1 occluded Integer (0,1,2,3) indicating occlusion state:
+ 0 = fully visible, 1 = partly occluded
+ 2 = largely occluded, 3 = unknown
+ 1 alpha Observation angle of object, ranging [-pi..pi]
+ 4 bbox 2D bounding box of object in the image (0-based index):
+ contains left, top, right, bottom pixel coordinates
+ 3 dimensions 3D object dimensions: height, width, length (in meters)
+ 3 location 3D object location x,y,z in camera coordinates (in meters)
+ 1 rotation_y Rotation ry around Y-axis in camera coordinates [-pi..pi]
+ 1 score Only for results: Float, indicating confidence in
+ detection, needed for p/r curves, higher is better.
+```
+
+Assume we use the Waymo dataset.
+After downloading the data, we need to implement a function to convert both the input data and annotation format into the KITTI style. Then we can implement WaymoDataset inherited from KittiDataset to load the data and perform training and evaluation.
+
+Specifically, we implement a waymo [converter](../tools/data_converter/waymo_converter.py) to convert Waymo data into KITTI format and a waymo dataset [class](../mmdet3d/datasets/waymo_dataset.py) to process it. Because we preprocess the raw data and reorganize it like KITTI, the dataset class could be implemented more easily by inheriting from KittiDataset. The last thing needed to be noted is the evaluation protocol you would like to use. Because Waymo has its own evaluation approach, we further incorporate it into our dataset class. Afterwards, users can successfully convert the data format and use `WaymoDataset` to train and evaluate the model.
+
+For more details about the intermediate results of preprocessing of Waymo dataset, please refer to its [tutorial](./tutorials/waymo.md).
+
+## Prepare a config
+
+The second step is to prepare configs such that the dataset could be successfully loaded. In addition, adjusting hyperparameters is usually necessary to obtain decent performance in 3D detection.
+
+Suppose we would like to train PointPillars on Waymo to achieve 3D detection for 3 classes, vehilce, cyclist and pedestrian, we need to prepare dataset config like [this](../mmdet3d/datasets/waymo_dataset.py), model config like [this](../configs/_base_/models/hv_pointpillars_secfpn_waymo.py) and combine them like [this](../configs/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class.py), compared to KITTI [dataset config](../configs/_base_/datasets/kitti-3d-3class.py), [model config](../configs/_base_/models/hv_pointpillars_secfpn_kitti.py) and [overall](../configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py).
+
+## Train a new model
+
+To train a model with the new config, you can simply run
+
+```shell
+python tools/train.py configs/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class.py
+```
+
+For more detailed usages, please refer to the [Case 1](1_exist_data_model.md).
+
+## Test and inference
+
+To test the trained model, you can simply run
+
+```shell
+python tools/test.py configs/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class.py work_dirs/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class/latest.pth --eval waymo
+```
+
+**Note**: To use Waymo evaluation protocol, you need to follow the [tutorial](tutorials/waymo.md) and prepare files related to metrics computation as official instructions.
+
+For more detailed usages for test and inference, please refer to the [Case 1](1_exist_data_model.md).