Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect checkpoint results #18

Open
girish1511 opened this issue Aug 11, 2023 · 12 comments
Open

Incorrect checkpoint results #18

girish1511 opened this issue Aug 11, 2023 · 12 comments

Comments

@girish1511
Copy link

girish1511 commented Aug 11, 2023 •
edited
Loading

Hi authors,

Thank you for your amazing work. I have been trying to verify the checkpoint results but unfortunately I'm getting some unexpected results.

`Car AP@0.70, 0.70, 0.70:
bbox AP:27.2850, 22.1567, 15.7245
bev AP:19.3694, 14.8054, 14.6355
3d AP:10.7381, 9.0909, 9.0909
aos AP:23.12, 18.91, 14.14
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:21.2188, 15.2842, 13.2074
bev AP:16.5121, 11.5316, 9.7222
3d AP:3.4522, 2.3132, 2.1522
aos AP:17.10, 12.06, 10.29
Car AP@0.70, 0.50, 0.50:
bbox AP:27.2850, 22.1567, 15.7245
bev AP:40.3225, 25.3175, 25.2042
3d AP:29.5763, 23.2644, 16.2642
aos AP:23.12, 18.91, 14.14
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:21.2188, 15.2842, 13.2074
bev AP:34.8517, 24.6689, 18.0450
3d AP:26.6140, 18.2151, 14.1654
aos AP:17.10, 12.06, 10.29
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00

2023-08-03 18:48:56,428 INFO ---- 2d box evaluation ----
2023-08-03 18:50:14,761 INFO Car AP_R40@0.70, 0.70, 0.70:
bbox AP:0.0000, 0.0000, 0.0000
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000`

Any idea as to where I could be going wrong with the evaluation code?
@chenyilun95
Copy link
Owner

It looks like you did not load the model weight well. Please check the output result txt first. And then check if the model weights are loaded well.

@Blume95
Copy link

Blume95 commented Dec 12, 2023

It looks like you did not load the model weight well. Please check the output result txt first. And then check if the model weights are loaded well.

Hello there, I also have the same problem these days. I use the default dsgn2.ymal config file and load your pre-trained models of DSGN2 Please explain a bit about how to load the model weights correctly. Thanks

@chenyilun95
Copy link
Owner

Hi, can you show me the inference results and the log? In my old environment the results should be correct. Note that there could be some warning during loading the weight (ignore it, you might refer #11)

@Blume95
Copy link

Blume95 commented Dec 12, 2023

Yes, I attach the evaluation log info below:

2023-12-11 14:42:12,065 INFO Start logging
2023-12-11 14:42:12,065 INFO CUDA_VISIBLE_DEVICES=ALL
2023-12-11 14:42:12,065 INFO eval output dir: /workspace/Dataset/dsgn2_ep58.pth.eval/eval/epoch_58/val/default
2023-12-11 14:42:12,065 INFO cfg_file /workspace/Dataset/DSGN2/configs/stereo/kitti_models/dsgn2.yaml
2023-12-11 14:42:12,065 INFO batch_size 1
2023-12-11 14:42:12,065 INFO workers 2
2023-12-11 14:42:12,065 INFO exp_name default
2023-12-11 14:42:12,065 INFO eval_tag default
2023-12-11 14:42:12,065 INFO max_waiting_mins 30
2023-12-11 14:42:12,065 INFO save_to_file True
2023-12-11 14:42:12,065 INFO ckpt /workspace/Dataset/dsgn2_ep58.pth
2023-12-11 14:42:12,065 INFO ckpt_id None
2023-12-11 14:42:12,065 INFO start_epoch 0
2023-12-11 14:42:12,065 INFO launcher none
2023-12-11 14:42:12,065 INFO tcp_port 18888
2023-12-11 14:42:12,065 INFO local_rank 0
2023-12-11 14:42:12,065 INFO set_cfgs None
2023-12-11 14:42:12,065 INFO trainval False
2023-12-11 14:42:12,065 INFO imitation 2d
2023-12-11 14:42:12,065 INFO cfg.ROOT_DIR: /workspace/Dataset/DSGN2
2023-12-11 14:42:12,065 INFO cfg.LOCAL_RANK: 0
2023-12-11 14:42:12,065 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2023-12-11 14:42:12,065 INFO
cfg.DATA_CONFIG = edict()
2023-12-11 14:42:12,065 INFO cfg.DATA_CONFIG.DATASET: StereoKittiDataset
2023-12-11 14:42:12,065 INFO cfg.DATA_CONFIG.DATA_PATH: ./data/kitti
2023-12-11 14:42:12,065 INFO cfg.DATA_CONFIG.FLIP: True
2023-12-11 14:42:12,065 INFO cfg.DATA_CONFIG.FORCE_FLIP: False
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [2, -30.4, -3, 59.6, 30.4, 1]
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.VOXEL_SIZE: [0.05, 0.05, 0.1]
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.STEREO_VOXEL_SIZE: [0.2, 0.2, 0.2]
2023-12-11 14:42:12,066 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val
2023-12-11 14:42:12,066 INFO
cfg.DATA_CONFIG.INFO_PATH = edict()
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl']
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl']
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.USE_VAN: True
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.USE_PERSON_SITTING: True
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.BOXES_GT_IN_CAM2_VIEW: False
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.GENERATE_CORNER_HEATMAP: False
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.CAT_REFLECT_DIM: False
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.TRAIN_DATA_AUGMENTOR: [{'NAME': 'gt_sampling', 'ratio': 0.9, 'USE_ROAD_PLANE': True, 'DB_INFO_PATH': ['stereo_kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:7', 'Pedestrian:7', 'Cyclist:7'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True, 'filter_occlusion_overlap': 0.7, 'remove_overlapped': True}, {'NAME': 'random_flip'}, {'NAME': 'random_crop', 'MIN_REL_X': 0, 'MAX_REL_X': 0, 'MIN_REL_Y': 1.0, 'MAX_REL_Y': 1.0, 'MAX_CROP_H': 320, 'MAX_CROP_W': 1248}, {'NAME': 'filter_truncated', 'AREA_RATIO_THRESH': None, 'AREA_2D_RATIO_THRESH': None, 'GT_TRUNCATED_THRESH': 0.98}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.1, 0.1]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.97, 1.03]}]
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.TEST_DATA_AUGMENTOR: [{'NAME': 'random_crop', 'MIN_REL_X': 0, 'MAX_REL_X': 0, 'MIN_REL_Y': 1.0, 'MAX_REL_Y': 1.0, 'MAX_CROP_H': 320, 'MAX_CROP_W': 1248}]
2023-12-11 14:42:12,066 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z']
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z']
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.05, 0.05, 0.1], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 40000, 'test': 40000}}]
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.BASE_CONFIG: ./configs/stereo/dataset_configs/kitti_dataset_fused.yaml
2023-12-11 14:42:12,066 INFO cfg.DATA_CONFIG.later_flip: True
2023-12-11 14:42:12,066 INFO
cfg.MODEL = edict()
2023-12-11 14:42:12,066 INFO cfg.MODEL.NAME: stereo
2023-12-11 14:42:12,066 INFO cfg.MODEL.SYNC_BN: True
2023-12-11 14:42:12,066 INFO
cfg.MODEL.BACKBONE_3D = edict()
2023-12-11 14:42:12,066 INFO cfg.MODEL.BACKBONE_3D.NAME: DSGN2Backbone
2023-12-11 14:42:12,066 INFO cfg.MODEL.BACKBONE_3D.maxdisp: 288
2023-12-11 14:42:12,066 INFO cfg.MODEL.BACKBONE_3D.downsample_disp: 4
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.GN: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.cat_img_feature: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.cat_right_img_feature: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.voxel_occupancy: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.front_surface_depth: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.num_3dconvs: 1
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.num_3dconvs_hg: 1
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.drop_psv: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.drop_psv_loss: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.squeeze_geo: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.geometry_volume_shift: 1
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.inv_smooth_psv: 0.1
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.inv_smooth_geo: 0.1
2023-12-11 14:42:12,067 INFO
cfg.MODEL.BACKBONE_3D.feature_backbone = edict()
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.type: ResNet
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.depth: 34
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.num_stages: 4
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.out_indices: [0, 1, 2, 3]
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.frozen_stages: -1
2023-12-11 14:42:12,067 INFO
cfg.MODEL.BACKBONE_3D.feature_backbone.norm_cfg = edict()
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.norm_cfg.type: BN
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.norm_cfg.requires_grad: True
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.norm_eval: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.style: pytorch
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.with_max_pool: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.deep_stem: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.block_with_final_relu: False
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.base_channels: 64
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.strides: [1, 2, 1, 1]
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.dilations: [1, 1, 2, 4]
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone.num_channels_factor: [1, 2, 2, 2]
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_backbone_pretrained: torchvision://resnet34
2023-12-11 14:42:12,067 INFO
cfg.MODEL.BACKBONE_3D.feature_neck = edict()
2023-12-11 14:42:12,067 INFO cfg.MODEL.BACKBONE_3D.feature_neck.GN: False
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.drop_psv: False
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.in_dims: [3, 64, 128, 128, 128]
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.start_level: 2
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.stereo_dim: [128, 96]
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.with_upconv: True
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.with_upconv_voxel: True
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.upconv_type: catk3
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.cat_img_feature: True
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.cat_right_img_feature: True
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.sem_dim: [128, 96]
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.feature_neck.with_sem_neck: True
2023-12-11 14:42:12,068 INFO
cfg.MODEL.BACKBONE_3D.sem_neck = edict()
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.type: FPN
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.in_channels: [96]
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.out_channels: 64
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.start_level: 0
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.add_extra_convs: on_output
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.sem_neck.num_outs: 5
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.cost_volume: [{'type': 'concat', 'downsample': 4}]
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.cv_dim: 32
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.rpn3d_dim: 32
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.downsampled_depth_offset: 0.5
2023-12-11 14:42:12,068 INFO cfg.MODEL.BACKBONE_3D.num_hg: 0
2023-12-11 14:42:12,068 INFO
cfg.MODEL.DENSE_HEAD_2D = edict()
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.NAME: MMDet2DHead
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.use_3d_center: True
2023-12-11 14:42:12,068 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg = edict()
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.type: ATSSAdvHead
2023-12-11 14:42:12,068 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.norm_cfg = edict()
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.norm_cfg.type: BN
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.norm_cfg.requires_grad: True
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.reg_class_agnostic: False
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.seperate_extra_reg_branch: False
2023-12-11 14:42:12,068 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.num_classes: 3
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.in_channels: 64
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.stacked_convs: 4
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.feat_channels: 64
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator.type: AnchorGenerator
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator.ratios: [1.0]
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator.octave_base_scale: 16
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator.scales_per_octave: 1
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.anchor_generator.strides: [4, 8, 16, 32, 64]
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.num_extra_reg_channel: 0
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.bbox_coder = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.bbox_coder.type: DeltaXYWHBBoxCoder
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.bbox_coder.target_means: [0.0, 0.0, 0.0, 0.0]
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.bbox_coder.target_stds: [0.1, 0.1, 0.2, 0.2]
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls.type: FocalLoss
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls.use_sigmoid: True
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls.gamma: 2.0
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls.alpha: 0.25
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_cls.loss_weight: 1.0
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.loss_bbox = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_bbox.type: GIoULoss
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_bbox.loss_weight: 2.0
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.loss_centerness = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_centerness.type: CrossEntropyLoss
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_centerness.use_sigmoid: True
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.loss_centerness.loss_weight: 1.0
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg = edict()
2023-12-11 14:42:12,069 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.assigner = edict()
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.assigner.type: ATSS3DCenterAssigner
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.assigner.topk: 9
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.allowed_border: -1
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.pos_weight: -1
2023-12-11 14:42:12,069 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.append_3d_centers: True
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.train_cfg.debug: False
2023-12-11 14:42:12,070 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg = edict()
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.nms_pre: 1000
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.min_bbox_size: 0
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.score_thr: 0.05
2023-12-11 14:42:12,070 INFO
cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.nms = edict()
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.nms.type: nms
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.nms.iou_threshold: 0.6
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD_2D.cfg.test_cfg.max_per_img: 100
2023-12-11 14:42:12,070 INFO
cfg.MODEL.MAP_TO_BEV = edict()
2023-12-11 14:42:12,070 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2023-12-11 14:42:12,070 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 160
2023-12-11 14:42:12,070 INFO cfg.MODEL.MAP_TO_BEV.SPARSE_INPUT: False
2023-12-11 14:42:12,070 INFO
cfg.MODEL.BACKBONE_2D = edict()
2023-12-11 14:42:12,070 INFO cfg.MODEL.BACKBONE_2D.NAME: HgBEVBackbone
2023-12-11 14:42:12,070 INFO cfg.MODEL.BACKBONE_2D.num_channels: 64
2023-12-11 14:42:12,070 INFO cfg.MODEL.BACKBONE_2D.GN: False
2023-12-11 14:42:12,070 INFO
cfg.MODEL.DENSE_HEAD = edict()
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.NAME: DetHead
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.NUM_CONVS: 2
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.GN: False
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.CLAMP_VALUE: 10.0
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.xyz_for_angles: True
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.hwl_for_angles: True
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.do_feature_imitation: False
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.imitation_cfg: [{'lidar_feature_layer': 'spatial_features_2d', 'stereo_feature_layer': 'spatial_features_2d', 'normalize': 'cw_scale', 'layer': 'conv2d', 'channel': 64, 'ksize': 1, 'use_relu': False, 'mode': 'inbox'}, {'lidar_feature_layer': 'volume_features', 'stereo_feature_layer': 'volume_features', 'normalize': 'cw_scale', 'layer': 'conv3d', 'channel': 32, 'ksize': 1, 'use_relu': False, 'mode': 'inbox'}]
2023-12-11 14:42:12,070 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 1, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 1, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 1, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2023-12-11 14:42:12,070 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2023-12-11 14:42:12,071 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER_CONFIG = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER_CONFIG.div_by_diagonal: True
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER_CONFIG.use_corners: False
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER_CONFIG.use_tanh: False
2023-12-11 14:42:12,071 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.REG_LOSS_TYPE: WeightedSmoothL1Loss
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.IOU_LOSS_TYPE: IOU3dLoss
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.IMITATION_LOSS_TYPE: WeightedL2WithSigmaLoss
2023-12-11 14:42:12,071 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 0.5
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.iou_weight: 1.0
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.imitation_weight: 1.0
2023-12-11 14:42:12,071 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2023-12-11 14:42:12,071 INFO
cfg.MODEL.DEPTH_LOSS_HEAD = edict()
2023-12-11 14:42:12,071 INFO
cfg.MODEL.DEPTH_LOSS_HEAD.LOSS_TYPE = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.DEPTH_LOSS_HEAD.LOSS_TYPE.ce: 1.0
2023-12-11 14:42:12,071 INFO cfg.MODEL.DEPTH_LOSS_HEAD.WEIGHTS: [1.0]
2023-12-11 14:42:12,071 INFO
cfg.MODEL.POST_PROCESSING = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2023-12-11 14:42:12,071 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: True
2023-12-11 14:42:12,071 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2023-12-11 14:42:12,072 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.25
2023-12-11 14:42:12,072 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2023-12-11 14:42:12,072 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2023-12-11 14:42:12,072 INFO
cfg.OPTIMIZATION = edict()
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 1
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 60
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.OPTIMIZER: adamw
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.LR: 0.001
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.0001
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [50]
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.LR_WARMUP: True
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2023-12-11 14:42:12,072 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2023-12-11 14:42:12,072 INFO cfg.TAG: dsgn2
2023-12-11 14:42:12,072 INFO cfg.EXP_GROUP_PATH: workspace_Dataset_DSGN2_configs_stereo_kitti_models
2023-12-11 14:42:12,105 INFO Loading KITTI dataset
2023-12-11 14:42:12,184 INFO Total samples for KITTI dataset: 3769
2023-12-11 14:42:13,657 INFO ==> (Stereo) Loading parameters from checkpoint /workspace/Dataset/dsgn2_ep58.pth to GPU
2023-12-11 14:42:14,916 INFO ==> Done (loaded 490/490)
2023-12-11 14:42:14,932 INFO *************** EPOCH 58 EVALUATION *****************
2023-12-11 15:01:21,123 INFO *************** Performance of EPOCH 58 *****************
2023-12-11 15:01:21,123 INFO Generate label finished(sec_per_example: 0.3041 second).
2023-12-11 15:01:21,124 INFO recall_roi_0.3: 0.000000
2023-12-11 15:01:21,124 INFO recall_rcnn_0.3: 0.123818
2023-12-11 15:01:21,124 INFO recall_roi_0.5: 0.000000
2023-12-11 15:01:21,124 INFO recall_rcnn_0.5: 0.104340
2023-12-11 15:01:21,124 INFO recall_roi_0.7: 0.000000
2023-12-11 15:01:21,124 INFO recall_rcnn_0.7: 0.035881
2023-12-11 15:01:21,124 INFO depth_error_all_local_median: 0.994070
2023-12-11 15:01:21,124 INFO depth_error_all_local_0.2m: 0.812193
2023-12-11 15:01:21,124 INFO depth_error_all_local_0.4m: 0.674763
2023-12-11 15:01:21,124 INFO depth_error_all_local_0.8m: 0.526444
2023-12-11 15:01:21,125 INFO depth_error_all_local_1.6m: 0.390836
2023-12-11 15:01:21,136 INFO depth_error_fg_local_statistics_perbox_err_median: 3.380206
2023-12-11 15:01:21,147 INFO depth_error_fg_local_statistics_perbox_err_0.2m: 0.760238
2023-12-11 15:01:21,154 INFO depth_error_fg_local_statistics_perbox_err_0.4m: 0.622323
2023-12-11 15:01:21,162 INFO depth_error_fg_local_statistics_perbox_err_0.8m: 0.505881
2023-12-11 15:01:21,170 INFO depth_error_fg_local_statistics_perbox_err_1.6m: 0.431304
2023-12-11 15:01:21,179 INFO Average predicted number of objects(3769 samples): 0.716
2023-12-11 15:01:21,507 INFO ---- 3d box evaluation ----
2023-12-11 15:01:34,395 INFO Car AP@0.70, 0.70, 0.70:
bbox AP:27.4231, 22.1915, 15.7245
bev AP:19.2701, 14.7917, 14.6417
3d AP:10.7169, 9.0909, 9.0909
aos AP:23.24, 18.95, 14.14
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:21.2600, 15.3158, 13.2495
bev AP:16.4342, 11.4518, 9.6679
3d AP:3.3553, 2.2894, 2.1223
aos AP:17.18, 12.10, 10.34
Car AP@0.70, 0.50, 0.50:
bbox AP:27.4231, 22.1915, 15.7245
bev AP:40.3457, 25.3265, 25.1971
3d AP:29.5156, 23.2514, 16.2745
aos AP:23.24, 18.95, 14.14
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:21.2600, 15.3158, 13.2495
bev AP:34.8907, 22.4836, 18.0368
3d AP:26.6725, 18.2056, 14.1679
aos AP:17.18, 12.10, 10.34
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
bev AP:0.0000, 0.0000, 0.0000
3d AP:0.0000, 0.0000, 0.0000
aos AP:0.00, 0.00, 0.00

2023-12-11 15:01:34,400 INFO ---- 2d box evaluation ----
2023-12-11 15:02:05,997 INFO Car AP_R40@0.70, 0.70, 0.70:
bbox AP:0.0000, 0.0000, 0.0000
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000

@chenyilun95
Copy link
Owner

Thanks!

  1. Can you check the KITTI output results like 000001.txt and let me know what the detected bounding boxes are inside?
  2. Did you match the pytorch and spconv environment? And some people get some problems with spconv version.
  3. Can you train the model in your machine and verify the results? I do not have the KITTI dataset on my machine right now so it takes time for me to check the pre-trained checkpoints.

@Blume95
Copy link

Blume95 commented Dec 13, 2023

Thanks for the info~

  1. Can you check the KITTI output results like 000001.txt and let me know what the detected bounding boxes are inside?

Nothing inside the default/final_result/data/000001.txt (No bounding boxes)

  1. Did you match the pytorch and spconv environment? And some people get some problems with spconv version.

addict 2.4.0
backcall 0.2.0
beautifulsoup4 4.9.3
certifi 2020.6.20
cffi 1.14.0
chardet 3.0.4
cmake 3.27.9
conda 4.9.1
conda-build 3.20.5
conda-package-handling 1.7.0
contourpy 1.1.1
cryptography 2.9.2
cycler 0.12.1
Cython 3.0.6
dataclasses 0.6
decorator 4.4.2
dnspython 2.0.0
easydict 1.11
filelock 3.0.12
fonttools 4.46.0
future 0.18.2
glob2 0.7
idna 2.9
imageio 2.33.1
importlib-metadata 7.0.0
importlib-resources 6.1.1
ipython 7.18.1
ipython-genutils 0.2.0
jedi 0.17.2
Jinja2 2.11.2
kiwisolver 1.4.5
lazy-loader 0.3
libarchive-c 2.9
liga 0.1.0+aee3731 /workspace/Dataset/LIGA-Stereo
llvmlite 0.36.0
MarkupSafe 1.1.1
matplotlib 3.7.4
mkl-fft 1.2.0
mkl-random 1.1.1
mkl-service 2.3.0
mmcv-full 1.4.0
mmdet 2.22.0 /workspace/Dataset/LIGA-Stereo/mmdetection_kitti
networkx 3.1
numba 0.53.0
numpy 1.22.0
olefile 0.46
opencv-python 4.8.1.78
packaging 23.2
parso 0.7.0
pcdet 0.1.0+a904f61 /workspace/Dataset/DSGN2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 10.1.0
pip 20.0.2
pkginfo 1.6.0
platformdirs 4.1.0
prompt-toolkit 3.0.8
protobuf 4.25.1
psutil 5.7.2
ptyprocess 0.6.0
pycocotools 2.0.7
pycosat 0.6.3
pycparser 2.20
Pygments 2.7.1
pyOpenSSL 19.1.0
pyparsing 3.1.1
PySocks 1.7.1
python-dateutil 2.8.2
python-etcd 0.4.5
pytz 2020.1
PyWavelets 1.4.1
PyYAML 5.3.1
requests 2.23.0
ruamel-yaml 0.15.87
scikit-image 0.21.0
scipy 1.10.1
setuptools 46.4.0.post20200518
six 1.14.0
soupsieve 2.0.1
spconv 1.2.1
tensorboardX 2.6.2.2
terminaltables 3.1.10
tifffile 2023.7.10
tomli 2.0.1
torch 1.7.0
torchelastic 0.2.1
torchvision 0.8.0
tqdm 4.46.0
traitlets 5.0.5
typing-extensions 3.7.4.3
urllib3 1.25.8
wcwidth 0.2.5
wheel 0.34.2
yapf 0.40.2
zipp 3.17.0
The pytorch version ==1.7.0 and the spconv version ==1.2.1

  1. Can you train the model in your machine and verify the results? I do not have the KITTI dataset on my machine right now so it takes time for me to check the pre-trained checkpoints.

I am training the model on one 3090ti with batch_size == 1, it still needs two days. I will update the result here.

@chenyilun95
Copy link
Owner

  1. Are there any detected bounding boxes in the 000xxx.txt?
  2. Cool, you can check the intermediate results like epoch-10, which should yield reasonable performance (>5 AP_CAR).

@Blume95
Copy link

Blume95 commented Dec 13, 2023

Are there any detected bounding boxes in the 000xxx.txt?

Yes, for example:

In 000004.txt: Car -1 -1 -7.4348 278.6960 178.8151 349.5790 215.2029 1.785065 1.764839 4.631009 -15.715478 2.122239 38.505917 -7.822333 0.21516302

Cool, you can check the intermediate results like epoch-10, which should yield reasonable performance (>5 AP_CAR).

I have tested the 34-epoch, the result likes below:

2023-12-13 10:45:00,225 INFO *************** Performance of EPOCH 34 *****************
2023-12-13 10:45:00,225 INFO Generate label finished(sec_per_example: 0.3142 second).
2023-12-13 10:45:00,225 INFO recall_roi_0.3: 0.000000
2023-12-13 10:45:00,225 INFO recall_rcnn_0.3: 0.853229
2023-12-13 10:45:00,225 INFO recall_roi_0.5: 0.000000
2023-12-13 10:45:00,226 INFO recall_rcnn_0.5: 0.753446
2023-12-13 10:45:00,226 INFO recall_roi_0.7: 0.000000
2023-12-13 10:45:00,226 INFO recall_rcnn_0.7: 0.493849
2023-12-13 10:45:00,226 INFO depth_error_all_local_median: 0.109514
2023-12-13 10:45:00,226 INFO depth_error_all_local_0.2m: 0.335280
2023-12-13 10:45:00,226 INFO depth_error_all_local_0.4m: 0.207142
2023-12-13 10:45:00,226 INFO depth_error_all_local_0.8m: 0.130046
2023-12-13 10:45:00,226 INFO depth_error_all_local_1.6m: 0.084545
2023-12-13 10:45:00,235 INFO depth_error_fg_local_statistics_perbox_err_median: 0.658179
2023-12-13 10:45:00,243 INFO depth_error_fg_local_statistics_perbox_err_0.2m: 0.518299
2023-12-13 10:45:00,249 INFO depth_error_fg_local_statistics_perbox_err_0.4m: 0.325645
2023-12-13 10:45:00,254 INFO depth_error_fg_local_statistics_perbox_err_0.8m: 0.191848
2023-12-13 10:45:00,259 INFO depth_error_fg_local_statistics_perbox_err_1.6m: 0.123333
2023-12-13 10:45:00,267 INFO Average predicted number of objects(3769 samples): 13.574
2023-12-13 10:45:00,619 INFO ---- 3d box evaluation ----
/opt/conda/lib/python3.8/site-packages/numba/core/typed_passes.py:326: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 160:
@numba.jit(nopython=True, parallel=True)
def d3_box_overlap_kernel(boxes, qboxes, rinc, criterion=-1):
^

warnings.warn(errors.NumbaPerformanceWarning(msg,
/opt/conda/lib/python3.8/site-packages/numba/core/typed_passes.py:326: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 160:
@numba.jit(nopython=True, parallel=True)
def d3_box_overlap_kernel(boxes, qboxes, rinc, criterion=-1):
^

warnings.warn(errors.NumbaPerformanceWarning(msg,
2023-12-13 10:45:27,724 INFO Car AP@0.70, 0.70, 0.70:
bbox AP:97.2287, 90.0781, 88.7225
bev AP:88.5526, 76.0166, 68.7856
3d AP:79.1395, 66.1320, 58.5258
aos AP:97.04, 89.78, 88.05
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.7335, 94.4128, 91.2468
bev AP:89.5465, 76.3046, 70.7628
3d AP:82.8323, 64.9919, 59.4407
aos AP:98.54, 94.05, 90.50
Car AP@0.70, 0.50, 0.50:
bbox AP:97.2287, 90.0781, 88.7225
bev AP:96.9713, 89.6054, 87.2113
3d AP:90.8074, 89.1699, 84.4519
aos AP:97.04, 89.78, 88.05
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.7335, 94.4128, 91.2468
bev AP:98.3775, 92.0581, 88.6882
3d AP:96.3894, 91.3967, 86.3852
aos AP:98.54, 94.05, 90.50
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:45.4138, 39.5464, 36.9842
bev AP:30.2508, 26.1992, 24.3483
3d AP:25.8574, 22.3619, 20.4178
aos AP:20.07, 17.86, 16.87
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:43.9603, 37.1875, 34.1949
bev AP:26.1509, 21.9390, 19.8038
3d AP:20.6679, 17.3777, 15.3087
aos AP:19.48, 16.83, 15.50
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:45.4138, 39.5464, 36.9842
bev AP:49.1159, 43.0023, 39.5303
3d AP:49.0571, 42.9293, 39.4092
aos AP:20.07, 17.86, 16.87
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:43.9603, 37.1875, 34.1949
bev AP:47.2879, 40.7037, 37.4303
3d AP:47.1765, 40.3084, 37.2881
aos AP:19.48, 16.83, 15.50
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:68.7559, 43.9899, 42.1120
bev AP:61.2625, 38.3186, 36.2334
3d AP:59.8401, 37.2659, 34.4511
aos AP:46.98, 30.67, 29.65
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:69.0910, 42.6764, 39.9248
bev AP:60.7892, 35.7985, 33.0912
3d AP:58.3736, 34.2494, 31.3953
aos AP:47.56, 29.96, 28.33
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:68.7559, 43.9899, 42.1120
bev AP:68.6062, 43.2530, 41.3012
3d AP:68.5538, 42.9903, 41.2211
aos AP:46.98, 30.67, 29.65
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:69.0910, 42.6764, 39.9248
bev AP:69.3643, 41.8688, 38.7695
3d AP:69.2159, 41.7177, 38.6152
aos AP:47.56, 29.96, 28.33

2023-12-13 10:45:27,732 INFO ---- 2d box evaluation ----
2023-12-13 10:46:00,480 INFO Car AP_R40@0.70, 0.70, 0.70:
bbox AP:0.0000, 0.0000, 0.0000
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.0000, 0.0000, 0.0000
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.0000, 0.0000, 0.0000

2023-12-13 10:46:00,480 INFO Result is save to /workspace/Dataset/DSGN2/outputs/workspace_Dataset_DSGN2_configs_stereo_kitti_models/dsgn2.default/ckpt/checkpoint_epoch_34.pth.eval/eval/epoch_34/val/default
2023-12-13 10:46:00,480 INFO Evaluation done.*

The 3D evaluation looks good, but why the performance of the 2D bounding box prediction is 0?

@chenyilun95
Copy link
Owner

Cool, the result looks normal.

The 2D bounding box results are referred to the box AP. For example,

Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.7335, 94.4128, 91.2468
bev AP:89.5465, 76.3046, 70.7628
3d AP:82.8323, 64.9919, 59.4407
aos AP:98.54, 94.05, 90.50

The 2D bbox AP_{0.7}_R40 (moderate) is 94.4, the BEV bbox AP_R40 is 76.3.

@sjg918
Copy link

sjg918 commented Jan 14, 2024

hello.
Thank you for your amazing work.
I'm having the exact same problem as @girish1511. @Blume95.
Below I upload the “xxxxxx.txt” result I got.
DSGN2-Output.zip
This result was obtained using the official checkpoint.
If you don't mind, can you provide reliable results (3769 txt files)?
There is no problem with training and I can create the output, but I would like to use official output.

THANKS !!

@chenyilun95
Copy link
Owner

Sorry that my machines have no the KITTI dataset right now. The official model is trained and inferenced with this code. Since you run the model, you can try training the model on your own and check the results after several epochs.

@moyanouo
Copy link

@chenyilun95 @Blume95 一样的问题...对官方的权重文件目前有解决的办法吗？或者有大佬可以提供一个训练过的文件吗？我只是想用测试一下结果，2080ti不太方便训练，谢谢！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@chenyilun95 @girish1511 @sjg918 @moyanouo @Blume95