[Fix] Updata bbox_3d info in cam_instances for mono3d-det task (#…

…2046)

* add mmengine assertion

* add docstring

* fix mminstall

* fix semantic segmentation configs

* fix semantic seg

* fix lint

* remove unused imports

* fix

* update pointnet2-s3dis config

* update data_list according to scene_idxs

* remove useless function

* fix bug lack `eval_ann_info` during evaluation

* fix bug

* update doc

* fix lint

* fix dataset converter

* update mmengine version

* delete whitespace

* [Fix] fix point cloud loop visualization error (#1914)

* fix point cloud loop visualization error

* fix browse_dataset

* fix browse_dataset

* support saving lidar_det

Co-authored-by: JingweiZhang12 <zjw18@mails.tsinghua.edu.cn>

* [Enhance] Show instance statistics before and after through pipeline (#1863)

* add instance statistics before and after through pipeline

* add docstring

* support showing cat-wise instance statistics

* show all statistics of the dataset

* small fix

* polish code

* show table

* small fix

* rename some varibles

* update task name and config for fcos3d++ on nus (#1927)

* update docs

* update docs

* Update det3d_dataset.py

* remove unnecessary optional docs

* update docs

* update docstrings

* update docs

* fix lint

* update docs

* [Refactor]: Refactor SASSD (#1901)

* refactor sassd

* rename some varibles

* add ut for sassd

* add some comments

* add some comments

* [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929)

* Use mmeval.MeanIoU

* fix comments and add self.reset

* solve some comments

* fix docstring

Co-authored-by: ChaimZhu <zhuchenming@pjlab.org.cn>

* fix

* fix

* fix

* fix lint

* add args and kwargs

* [Docs] Add mmyolo link in README (#1935)

* [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929)

* Use mmeval.MeanIoU

* fix comments and add self.reset

* add mmyolo

Co-authored-by: ChaimZhu <zhuchenming@pjlab.org.cn>

* [Fix] fix seg metric issues and circle ci (#1962)

* fix circle ci

* update

* update

* [Enhance] Support evaluation for waymo dataset on s3 filesystem when tensorflow>=2.6.0 (#1958)

* support tensorflow>2.6.0

* add filter conditions

* [Fix]: change --pkl to --pkl-path in the data upgrade scripts (#1965)

* [Refactor] update `metainfo` in pkl file and add `categories` into `metainfo` (#1934)

* update metainfo in pkl file

* update

* add version for waymo

* update kitti metric

* [Refactor] Refactor FCAF3D (#1945)

* add fcaf3d config

* support inference of fcaf3d on scannet-3d

* minor changes

* fix config of scannet

* align test precision of facaf3d && support training

* add ut

* fix bugs of dataset conversion and indoor_metric

* support fcaf3d on sunrgbd datasets and add rorate_iou_loss ut

* small fix

* add docstring and typehint

* add typehint for SparseTensor

* remove pdb

* fix fcaf3d ut

* [Docs] Update Chinese documentation (#1891)

* [Enhance] Add an example of pcd to bin format. (#1973)

* [Enhance] Add an example of pcd to bin format.

* Fix installation and typo.

* [Fix]: fix image conversion of Waymo to avoid information loss (#1979)

* add mmeval

* [Refactor] rename `CLASSES` and `PALETTE` to `classes` and `palette` in dataset metainfo (#1932)

* rame CLASS and PALETTE to class and palette

* change mmcv-full to mmcv

* fix comments

* [CI]: fix CI (#2007)

* Update docs && support loading old checkpoints in FCAF3D  (#1974)

* add loading converted keys and README, metafile

* update readme

* reorganize readme

* add version in FCAF3d detector

* fix converter error

* [Fix] fix kitti evaluation bugs on test dataset (#2005)

* fix kitti test evaluation bugs

* update

* [Fix] update the key name of modules from mmdet in cfgs (#2011)

* [Fix] fix instance statistics when only detecting a single class (#2003)

* Update iou_neg_piecewise_sampler.py

* fix lint

* [Enhance] Remove useless json file converter (#1971)

* [Enhance] Refactor the import of `mmdet` (#1947)

* fix config

* refactor import of `mmdet`

* fix lint

* fix lint

* [Enhance] Speed up evaluation on waymo (#2008)

* support fast eval on waymo

* support waymo evaluatioin more flexible and faster

* support waymo evaluatioin more flexible and faster

* renames

* add docs

* add guides for multi-thread evaluation toolkit

* fix docstring

* add download link for idx2metainfo

* add docstring

* set convert_kitti_format=False in Lidar-based methods

* fix docs

* add docstring

* [Features] Support PV_RCNN modules (#1957)

* add pvrcnn module code

* add voxelsa

* fix

* fix comments

* fix comments

* fix comments

* add stack sa

* fix

* fix comments

* fix comments

* fix

* add ut

* fix comments

* fix kitti mono_det bug

* [Fix]Fix a bug in StackQueryAndGroup (#2043)

* fix a bug

* fix a batch inference bug

* fix docs

* [Community] Add 'Projects/' folder, and the first example project (#2042)

* add project in mmdet3d

* fix

* fix

* [Fix] Fix waymo converter (#2040)

* fix waymo converter

* [Fix] Update waymo converter and fix lint

* Update waymo_converter.py

* Update kitti_converter.py

* Update update_infos_to_v2.py

* revert

* fix paths in `update_infos_to_v2` on Windows by using `pathlib` (#2031)

* fix paths on Windows by using `pathlib`

* refactor Path import

* simplify the warning and error when using open3d and MinkowskiEngine (#2027)

* [FIX] replace DefaultFormatBundle/3D with Pack(3D)DetInputs (#1987)

* replace defaultformatbundle3d with pack3ddetinputs

* remove normalize, pad, imagetotensor from configs

* rm unused key 'img_norm_cfg'

* fix lint errors

* fix lint errors

* fix lint error

* xx

* [Docs] Refine the documentation (#1994)

* refine doc

* refine docs

* replace `CLASSES` with `classes`

* update doc

* Minor fix

Co-authored-by: Tai-Wang <tab_wang@outlook.com>

* Update converter

* [Fix] Fix some loading bugs and support fov_image_based mode in Waymo dataset.  (#1942)

* modify sample_id to sample_id and support fov_image_based on waymo dataset

* Update waymo_metric.py

* Minor fix

* Minor fix

* Minor fix

* Minor fix

* Minor fix

* Minor fix

* Minor fixes

* Minor fixes

* Remove optional

* fix dataset instances converting bugs

* Add a blank line to fix the doc compilation format

* Fix the bin file name in waymo_fov config

* Resolve conflicts

* fix ci and other things

Co-authored-by: Tai-Wang <tab_wang@outlook.com>
Co-authored-by: lianqing11 <lianqing11@foxmail.com>
Co-authored-by: ChaimZhu <zhuchenming@pjlab.org.cn>

* [Fix] Add num_features in Lyft dataset info file (#1948)

Co-authored-by: Tai-Wang <tab_wang@outlook.com>

* [Feature] Support FCAF3D on S3DIS dataset in `dev-1.x`  branch (#1984)

* support fcaf3d for s3dis dataset

* Update convert_utils.py

* Update seg3d_dataset.py

* Delete compose.py

* fix import error

* use `mmengine.Compose`

* Update s3dis-3d.py

* Update fcaf3d_2xb8_s3dis-3d-5class.py

* Update s3dis_dataset.py

* update unittest for s3dis

* update docs

* use `mmcv.Compose` instead of `mmengine.Compose`

* update docstring

* fix s3dis preprocessing bug

* Add typehint

* Update config and fix s3dis dataset

* update typehit

* Update convert_utils.py

* Update README and metafile

Co-authored-by: Tai-Wang <tab_wang@outlook.com>

* Update kitti_dataset.py

* replace `List` wiith `Tuple` format to represent `origin`

* Update convert_utils.py

* add comments

* update

* resolve

Co-authored-by: ChaimZhu <zhuchenming@pjlab.org.cn>
Co-authored-by: JingweiZhang12 <zjw18@mails.tsinghua.edu.cn>
Co-authored-by: Qing Lian <lianqing1997@gmail.com>
Co-authored-by: Shaun <cq.xiang@foxmail.com>
Co-authored-by: Wencheng Wu <41542251+274869388@users.noreply.github.com>
Co-authored-by: Erli Ouyang <ouyangerli@gmail.com>
Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com>
Co-authored-by: vavanade <vavanade@users.noreply.github.com>
Co-authored-by: Tai-Wang <tab_wang@outlook.com>
Co-authored-by: lianqing11 <lianqing11@foxmail.com>

mmdet3d/datasets/convert_utils.py

@@ -253,16 +253,17 @@ def get_kitti_style_2d_boxes(info: dict,
  # transform the center from [0.5, 1.0, 0.5] to [0.5, 0.5, 0.5]
  dst = np.array([0.5, 0.5, 0.5])
  src = np.array([0.5, 1.0, 0.5])
- loc = loc + dim * (dst - src)
- loc_3d = np.copy(loc)
- gt_bbox_3d = np.concatenate([loc, dim, rot], axis=1).astype(np.float32)
+ # gravity center
+ loc_center = loc + dim * (dst - src)
+ gt_bbox_3d = np.concatenate([loc_center, dim, rot],
+ axis=1).astype(np.float32)
 
  # Filter out the corners that are not in front of the calibrated
  # sensor.
  corners_3d = box_np_ops.center_to_corner_box3d(
  gt_bbox_3d[:, :3],
  gt_bbox_3d[:, 3:6],
- gt_bbox_3d[:, 6], [0.5, 0.5, 0.5],
+ gt_bbox_3d[:, 6], (0.5, 0.5, 0.5),
  axis=1)
  corners_3d = corners_3d[0].T # (1, 8, 3) -> (3, 8)
  in_front = np.argwhere(corners_3d[2, :] > 0).flatten()
@@ -291,12 +292,12 @@ def get_kitti_style_2d_boxes(info: dict,
 
  # If mono3d=True, add 3D annotations in camera coordinates
  if mono3d and (repro_rec is not None):
+ # use bottom center to represent the bbox_3d
  repro_rec['bbox_3d'] = np.concatenate(
- [loc_3d, dim, rot],
- axis=1).astype(np.float32).squeeze().tolist()
+ [loc, dim, rot], axis=1).astype(np.float32).squeeze().tolist()
  repro_rec['velocity'] = -1 # no velocity in KITTI
 
- center_3d = np.array(loc).reshape([1, 3])
+ center_3d = np.array(loc_center).reshape([1, 3])
  center_2d_with_depth = points_cam2img(
  center_3d, camera_intrinsic, with_depth=True)
  center_2d_with_depth = center_2d_with_depth.squeeze().tolist()

mmdet3d/datasets/kitti_dataset.py

@@ -24,6 +24,15 @@ class KittiDataset(Det3DDataset):
  Defaults to dict(use_lidar=True).
  default_cam_key (str): The default camera name adopted.
  Defaults to 'CAM2'.
+ load_type (str): Type of loading mode. Defaults to 'frame_based'.
+
+ - 'frame_based': Load all of the instances in the frame.
+ - 'mv_image_based': Load all of the instances in the frame and need
+ to convert to the FOV-based data type to support image-based
+ detector.
+ - 'fov_image_based': Only load the instances inside the default
+ cam, and need to convert to the FOV-based data type to support
+ image-based detector.
  box_type_3d (str): Type of 3D box of this dataset.
  Based on the `box_type_3d`, the dataset will encapsulate the box
  to its original format then converted them to `box_type_3d`.
@@ -32,15 +41,6 @@ class KittiDataset(Det3DDataset):
  - 'LiDAR': Box in LiDAR coordinates.
  - 'Depth': Box in depth coordinates, usually for indoor dataset.
  - 'Camera': Box in camera coordinates.
- load_type (str): Type of loading mode. Defaults to 'frame_based'.
-
- - 'frame_based': Load all of the instances in the frame.
- - 'mv_image_based': Load all of the instances in the frame and need
- to convert to the FOV-based data type to support image-based
- detector.
- - 'fov_image_based': Only load the instances inside the default
- cam, and need to convert to the FOV-based data type to support
- image-based detector.
  filter_empty_gt (bool): Whether to filter the data with empty GT.
  If it's set to be True, the example with empty annotations after
  data pipeline will be dropped and a random example will be chosen

mmdet3d/datasets/waymo_dataset.py

@@ -156,20 +156,12 @@ def parse_ann_info(self, info: dict) -> dict:
  centers_2d = np.zeros((0, 2), dtype=np.float32)
  depths = np.zeros((0), dtype=np.float32)
 
- if self.load_type in ['fov_image_based', 'mv_image_based']:
- gt_bboxes_3d = CameraInstance3DBoxes(
- ann_info['gt_bboxes_3d'],
- box_dim=ann_info['gt_bboxes_3d'].shape[-1],
- origin=(0.5, 0.5, 0.5))
-
- else:
- # in waymo, lidar2cam = R0_rect @ Tr_velo_to_cam
- # convert gt_bboxes_3d to velodyne coordinates with `lidar2cam`
- lidar2cam = np.array(
- info['images'][self.default_cam_key]['lidar2cam'])
- gt_bboxes_3d = CameraInstance3DBoxes(
- ann_info['gt_bboxes_3d']).convert_to(self.box_mode_3d,
- np.linalg.inv(lidar2cam))
+ # in waymo, lidar2cam = R0_rect @ Tr_velo_to_cam
+ # convert gt_bboxes_3d to velodyne coordinates with `lidar2cam`
+ lidar2cam = np.array(info['images'][self.default_cam_key]['lidar2cam'])
+ gt_bboxes_3d = CameraInstance3DBoxes(
+ ann_info['gt_bboxes_3d']).convert_to(self.box_mode_3d,
+ np.linalg.inv(lidar2cam))
  ann_info['gt_bboxes_3d'] = gt_bboxes_3d
 
  anns_results = dict(