Bounding box orientation incorrect on custom nuscenes formatted dataset

[pkamat95]

Hi,

We've been using mmdetection3d to train a custom dataset that we've created that is in the nuscenes format, using our own classes and data. We can render the ground truth annotations using the nuscenes devkit, with nuscenes.render_sample_data(), and this visualisation is correct.

We altered nuscenes_converter.py to work with our dataset. As there were several instances of things being hard coded to work specifically with the nuscenes dataset, we had to change several things to get the converter working with our dataset.

We can use an altered version of browse_datasets.py to visualise the ground truth annotations after the conversion. When we visualise these, the translation is correct, but the orientation of the box is incorrect. Since translation appears perfectly, this leads me to think that there is no issue with the ego or sensor rotation and that it's a problem with only the annotation orientation. We can also train on the dataset and get a high mAP, but the predictions, as expected, have the same orientation as the ground truth data when visualised using mono_det_demo.py

After spending a lot of time attempting to understand where this rotation issue is coming from, we're at a loss. Hopefully someone could offer some help with this issue. @ZwwWayne I see that you've done this PR which seems quite relevant to what we've been working on. Perhaps you have an idea of what is going on?

I have packaged a dataset with a few images, and the conversion and visualisation scripts we've been using.
mmdetection3d.zip

If someone gets the time to help us out with this, it would be very appreciated.

If you want to run the conversion and visualisation scripts, build and run the docker container using the latest repo, and use the data folder in the attached zip as the DATA_DIR in docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection3d/data mmdetection3d. Then place the scoville folder from the attached zip in /mmdetection/ in the docker instance. Dataset conversion and visualisation can then be done using the following commands:

• python scoville/pencil_nuscenes_converter.py data/nuscenes_pencil_example/ data/nuscenes_pencil_example/
• python scoville/browse_dataset_pencil.py scoville/pencil_finetune.py --output-dir visualisations --task mono-det

[Tai-Wang]

Sorry for the little late reply. Awesome attempt! It's really interesting.

As you mentioned, we have made a refactoring in terms of the hack of yaw angle and dimensions for monocular 3D detection on nuScenes. After that PR, the data annotation files need to be re-generated and the models should also be re-trained. Have you done this and use the latest master?

[pkamat95]

Hi @Tai-Wang! Thanks for getting back to me. We're using the latest master and have integrated the part of the code that was refactored in the nuscenes conversion script into our conversion script, then regenerated the data annotation files. We still get the same issue when displaying ground truth.

It was the changes from tools/data_converter/nuscenes_converter.py that were made in this PR that needed to be made in the conversion script right?

[Tai-Wang]

If you have involved all the other refactoring in that PR, the overall implementation of mmdet3d should be correct. Then how about your mAOE in the evaluation? Does it also look good? If yes, the problem is only likely related to the visualization functions.

[bml1g12]

We had the same thought given we can get good mAP it seems the model and raw data is fine, but that the visualisation is somehow wrong. We have spent many days trying to debug what the issue with the visualisation is but every idea we had has failed, we'd be very grateful if you had any time or ideas to help us debug this one.

[Tai-Wang]

I mean the mAOE instead of mAP because the mAP of nuScenes does not consider the orientation of boxes (only consider whether the distance of centers is under certain thresholds).

[bml1g12]

Aha, I didn't realise this (I assumed incorrectly aligned boxes would produce poor mAP, similar to the 2D case with a low IoU threshold).

img_bbox_NuScenes/pencil_AP_dist_0.5: 0.8536, img_bbox_NuScenes/pencil_AP_dist_1.0: 0.8536, img_bbox_NuScenes/pencil_AP_dist_2.0: 0.8536, img_bbox_NuScenes/pencil_AP_dist_4.0: 0.8536, img_bbox_NuScenes/pencil_trans_err: 0.1198, img_bbox_NuScenes/pencil_scale_err: 0.9782, img_bbox_NuScenes/pencil_orient_err: 2.0613, img_bbox_NuScenes/pencil_vel_err: 1.0000, img_bbox_NuScenes/pencil_attr_err: 1.0000, img_bbox_NuScenes/mATE: 0.1198, img_bbox_NuScenes/mASE: 0.9782, img_bbox_NuScenes/mAOE: 2.0613, img_bbox_NuScenes/mAVE: 1.0000, img_bbox_NuScenes/mAAE: 1.0000, img_bbox_NuScenes/NDS: 0.5170, img_bbox_NuScenes/mAP: 0.8536

I am not too familiar with this metric: img_bbox_NuScenes/mAOE: 2.0613 but if I understand correctly it would suggest that the orientation is on average about 2 radians (118 degrees) out?

If so that helps us a lot, it would guess that it suggests that the visualization is not the issue after all, but in fact there is something wrong the with the orientation of ground truth data being fed in mmdet library such that the model is learning to provide inappropriate orientation.

The mystery still remains though as to why we can visualise the raw nuscenes ground data fine with the NuScenes visualiser. It would suggest there is a bug somewhere in our nuscenes converter I think - can you spot anything that looks suspicious?

[Tai-Wang]

Yes, your evaluation results show that your predicted orientations have something wrong. The mAP is defined based on the distance thresholds instead of 3D-IoU on nuScenes, so it can not reflect whether your orientation estimation is good or not.
So is there any update about the localization of your problem?

[bml1g12]

We haven't been able to spot where things are going wrong in our converter. We built our current converter using an older version of nuscenes_converter.py as the baseline, then adding in manually the latest code changes such as your PR to the converter we had written. I think over the next two weeks @pkamat95 will take a go at re-writing our nuscenes converter using the latest commit of 3DMPPE instead, in those hope that we just made a subtle mistake somewhere.

[bml1g12]

Tried rewriting our converter using the latest version as a starting point, but ended with the same issue :( Sadly we might have to abandon this unless anyone has any further ideas on how we could debug.

[VeeranjaneyuluToka]

Is there any solution for this? I am facing exactly same issue. When my yaw angle is zero, it works fine but if there is yaw angle, the visualization goes for toss as showed above. Would be great help if there is some solution for this. Thanks!

[VeeranjaneyuluToka]

FYI, I fixed this issue. I have taken yaw = -yaw While converting from json files to .pkl file, when i took yaw without any changes, the GTs are getting drawn as expected. So this is basically a change in my converting script, but predictions are in wrong orientation while running inference as i have raised the same in #1287.

[VeeranjaneyuluToka]

In my case, predictions are wrong becuase my training set samples are wrong.