About Robust Assessment

[haichen-ber]

Thank you for your work and open source code. I would like to know how the mAP ratio of KITTI is calculated in the article? Is it just the relative decrease ratio of AP@R11 0.7 0.7 0.7 of the car category? Or is there another way to calculate it? Below are the vanilla inference results I got about pointpillar and the inference results I got under PGD attack, can you give an example of mAP ratio calculation?

[Eaphan]

We use the mAP, which is the average AP@R40 0.7 0.7 0.7 (3d) on three classes for kitti dataset.
mAP ratio = mAP under attack/ mAP without attack.

[haichen-ber]

We use the mAP, which is the average AP@R40 0.7 0.7 0.7 (3d) on three classes for kitti dataset. mAP ratio = mAP under attack/ mAP without attack.

Thank you for your answer. I cannot calculate the results given in the paper based on the formula you gave. And AP@R40 0.7 0.7 0.7 only has the car category in the output results.

Here are the weights for the pointpillar I use:

[Eaphan]

Sorry for the confusion.

AP@R40 0.7 0.7 0.7 (3d) for car

AP@R40 0.5 0.5 0.5 (3d) for pedestrian and cyclist.

Then mAP is the average of them (9 values).

If you cannot calculate the results given in the paper, please check the parameters used in the script like key, eps, etc.

[haichen-ber]

Sorry for the confusion.

AP@R40 0.7 0.7 0.7 (3d) for car

AP@R40 0.5 0.5 0.5 (3d) for pedestrian and cyclist.

Then mAP is the average of them (9 values).

If you cannot calculate the results given in the paper, please check the parameters used in the script like key, eps, etc.

tks, I can reproduce it!