Empty ground truth boxes for training 3DSSD on Pedestrian KITTI

[zye1996]

Hi, I am trying to train 3DSSD on KITTI for Pedestrian objects by replacing class-related setting in config/3dssd/3dssd_kitti-3d-car.py. The training can proceed for some time and it stops with an error below after a couple epochs:

Traceback (most recent call last):
  File "tools/train.py", line 166, in <module>
    main()
  File "tools/train.py", line 162, in main
    meta=meta)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection/mmdet/apis/train.py", line 150, in train_detector
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
    self.run_iter(data_batch, train_mode=True)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
    **kwargs)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection/mmdet/models/detectors/base.py", line 246, in train_step
    losses = self(**data)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/models/detectors/base.py", line 59, in forward
    return self.forward_train(**kwargs)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/models/detectors/votenet.py", line 60, in forward_train
    bbox_preds, *loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
  File "/home/yzy/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func
    return old_func(*args, **kwargs)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/models/dense_heads/ssd_3d_head.py", line 145, in loss
    bbox_preds)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/models/dense_heads/ssd_3d_head.py", line 271, in get_targets
    seed_points)
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection/mmdet/core/utils/misc.py", line 26, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/models/dense_heads/ssd_3d_head.py", line 340, in get_targets_single
    gt_corner3d = gt_bboxes_3d.corners
  File "/home/yzy/PycharmProjects/AutoDrive/mmdetection3d/mmdet3d/core/bbox/structures/lidar_box3d.py", line 70, in corners
    assert len(self.tensor) != 0
AssertionError

It seems it is related to dataset sampler when sampling point cloud data without ground truth boxes. I am wondering if there is any setting for the sampler that can fix the error? I already tried to modify the part for data sampler as follow:

db_sampler = dict(
    data_root=data_root,
    info_path=data_root + 'kitti_dbinfos_train.pkl',
    rate=1.0,
    prepare=dict(filter_by_difficulty=[-1], filter_by_min_points=dict(Pedestrian=1)),
    classes=class_names,
    sample_groups=dict(Pedestrian=6))

[encore-zhou]

@zye1996 Hi, do you modify the class_names respectively?

[zye1996]

@zye1996 Hi, do you modify the class_names respectively?

Yes I did, here is my config with parts I modified, and it did train for around 10+ epoch before the error kicked in

_base_ = [
    '../_base_/models/3dssd.py', '../_base_/datasets/kitti-3d-pedestrian.py',
    '../_base_/default_runtime.py'
]

# dataset settings
dataset_type = 'KittiDataset'
data_root = 'data/kitti/'
class_names = ['Pedestrian']
point_cloud_range = [0, -40, -5, 70, 40, 3]
input_modality = dict(use_lidar=True, use_camera=False)
db_sampler = dict(
    data_root=data_root,
    info_path=data_root + 'kitti_dbinfos_train.pkl',
    rate=1.0,
    prepare=dict(filter_by_difficulty=[-1], filter_by_min_points=dict(Pedestrian=1)),
    classes=class_names,
    sample_groups=dict(Pedestrian=6))

file_client_args = dict(backend='disk')
# Uncomment the following if use ceph or other file clients.
# See https://mmcv.readthedocs.io/en/latest/api.html#mmcv.fileio.FileClient
# for more details.
# file_client_args = dict(
#     backend='petrel', path_mapping=dict(data='s3://kitti_data/'))

[encore-zhou]

Fix in #258

[RolandoAvides]

@zye1996 Did you manage to train 3DSSD on KITTI for the pedestrian class?

[jlqzzz]

@RolandoAvides @zye1996
Did you manage to train 3DSSD on KITTI for the pedestrian class?

[RolandoAvides]

Yes

[jlqzzz]

Yes

@RolandoAvides

Can you share your successful implementation?

At present, based on 3dssd-pytorch-openpcdet, only the accuracy of Car is close to normal, and the accuracy of pedestrian and cyclist is very very low!

I don't know how to solve it, please help me!

[zye1996]

@RolandoAvides @zye1996
Did you manage to train 3DSSD on KITTI for the pedestrian class?

Email me through zhenyiye1996@gmail.com since I made my repository privately.

[jlqzzz]

@RolandoAvides @zye1996
Did you manage to train 3DSSD on KITTI for the pedestrian class?

Email me through zhenyiye1996@gmail.com since I made my repository privately.

@zye1996

I sent you an email. Thank you very much!