Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhance] Add waymo unit test #455

Merged
merged 11 commits into from
Apr 30, 2021
Merged

Conversation

Wuziyi616
Copy link
Contributor

Fix issue#447. I add some basic unit tests for WaymoDataset. Mainly about getitem and show (because I want to check its compatibility in dataset_browse). I am not familiar with Waymo format of box and its evaluate function. Maybe someone can help me continue this PR?

@codecov
Copy link

codecov bot commented Apr 16, 2021

Codecov Report

Merging #455 (2e55bb6) into master (b00d3dc) will increase coverage by 0.11%.
The diff coverage is n/a.

❗ Current head 2e55bb6 differs from pull request most recent head 07d0933. Consider uploading reports for the commit 07d0933 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master     #455      +/-   ##
==========================================
+ Coverage   50.55%   50.67%   +0.11%     
==========================================
  Files         189      189              
  Lines       14453    14453              
  Branches     2366     2366              
==========================================
+ Hits         7307     7324      +17     
+ Misses       6655     6636      -19     
- Partials      491      493       +2     
Flag Coverage Δ
unittests 50.67% <ø> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmdet3d/datasets/waymo_dataset.py 19.17% <0.00%> (+8.80%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b00d3dc...07d0933. Read the comment docs.

@ZwwWayne
Copy link
Collaborator

The file sizes for unit tests are too big. We need to make the size of test data smaller. See other files for unit test.

@Wuziyi616
Copy link
Contributor Author

The file sizes for unit tests are too big. We need to make the size of test data smaller. See other files for unit test.

Yes I have made them smaller (each <1MB).

@Wuziyi616
Copy link
Contributor Author

Wuziyi616 commented Apr 19, 2021

I have some trouble in the Waymo protocol testing of test_evaluate() in test_waymo_dataset.py. I got expected mAP in the KITTI protocol, but wrong results in the Waymo protocol (all 0). The GT bbox I generated in gt.bin is the same as the predicted bbox after kitti_to_waymo conversion (here). I don't know why the results are wrong...

The predicted bbox after conversion:

objects {
  object {
    box {
      center_x: 69.67650333936074
      center_y: 33.340902794888315
      center_z: 0.7713991550544462
      width: 2.009999990463257
      length: 4.360000133514404
      height: 1.4600000381469727
      heading: -1.4807963232186179
    }
    type: TYPE_VEHICLE
  }
  score: 0.5
  context_name: "1071392229495085036_1844_790_1864_790"
  frame_timestamp_micros: 1507315488219118
}

The loaded gt.bin file GT bbox:

objects {
  object {
    box {
      center_x: 69.67650333936074
      center_y: 33.340902794888315
      center_z: 0.7713991550544462
      width: 2.009999990463257
      length: 4.360000133514404
      height: 1.4600000381469727
      heading: -1.4807963232186179
    }
    type: TYPE_VEHICLE
  }
  score: 1.0
  context_name: "1071392229495085036_1844_790_1864_790"
  frame_timestamp_micros: 1507315488219118
  camera_name: FRONT
}

@Tai-Wang
Copy link
Member

I have some trouble in the Waymo protocol testing of test_evaluate() in test_waymo_dataset.py. I got expected mAP in the KITTI protocol, but wrong results in the Waymo protocol (all 0). The GT bbox I generated in gt.bin is the same as the predicted bbox after kitti_to_waymo conversion (here). I don't know why the results are wrong...

The predicted bbox after conversion:

objects {
  object {
    box {
      center_x: 69.67650333936074
      center_y: 33.340902794888315
      center_z: 0.7713991550544462
      width: 2.009999990463257
      length: 4.360000133514404
      height: 1.4600000381469727
      heading: -1.4807963232186179
    }
    type: TYPE_VEHICLE
  }
  score: 0.5
  context_name: "1071392229495085036_1844_790_1864_790"
  frame_timestamp_micros: 1507315488219118
}

The loaded gt.bin file GT bbox:

objects {
  object {
    box {
      center_x: 69.67650333936074
      center_y: 33.340902794888315
      center_z: 0.7713991550544462
      width: 2.009999990463257
      length: 4.360000133514404
      height: 1.4600000381469727
      heading: -1.4807963232186179
    }
    type: TYPE_VEHICLE
  }
  score: 1.0
  context_name: "1071392229495085036_1844_790_1864_790"
  frame_timestamp_micros: 1507315488219118
  camera_name: FRONT
}

Sometimes evaluation methods of these datasets may not deal with all the corner cases. You may try to validate it with more samples or add some noises to these predictions.

@Wuziyi616
Copy link
Contributor Author

I finally solved the problem and this PR is ready for review now @ZwwWayne!

@Wuziyi616 Wuziyi616 requested a review from Tai-Wang April 28, 2021 06:13
@ZwwWayne ZwwWayne merged commit 951b521 into open-mmlab:master Apr 30, 2021
@Wuziyi616 Wuziyi616 deleted the waymo_unit_test branch April 30, 2021 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add unit test for Waymo dataset
3 participants