Substandard performance without postprocessing.

[MayankSingal]

Good Evening,

There seems to be lot of rule based post processing being done after getting the model outputs. (Forcing prediction to be snowboarding if snowboard is present etc.). Can you report the numbers without such post processing being applied? Evaluating the model after removing post-processing is giving me very bad results, so I'm not sure if I'm doing it properly. I couldn't find any mention of such numbers in the BMVC paper.

Thank you!

[gaochen315]

Thanks for the question.

We follow the processing steps by prior work (Visual Semantic Role Labeling, Detecting and Recognizing Human-Object Interactions) to remove prediction scores for certain action-object pairs. You can see in the eq2 of Visual Semantic Role Labeling, C is predefined. Thus, we know the object categories that are related to a specific action class.

You can simply add a --prior_flag 0 flag to the Test_ResNet_VCOCO.py, i.e. using the following command

python tools/Test_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000 --prior_flag 0

to obtain the mAP without any post-processing.

[MayankSingal]

Thank you for the quick response!

I ran the testing pipeline with the above command using prior_flag = 0.

On the pre-trained model provided, the mAP values obtained with prior flag = 0 are 2.38(Scenario 1) and 4.81(Scenario 2). Is such a huge drop expected? Compared to the reported mAP of 45, these are very surprising numbers.

Is the contribution of this prior knowledge 'C' so significant?

Thank you!

[MayankSingal]

Good Morning,
Any information on this?

Thank you!

[gaochen315]

As I recall, we will lose around 3mAP without post-processing. The mAP should be around 42.

I will check and see where is the problem.

[ZHUXUHAN]

Good Morning,
Any information on this?

Thank you!

hi, man,do you know the metric for hico-det

Good Morning,
Any information on this?

Thank you!

hi, man did you test the vcoco or the hico-det datasets, and how did you test the model, I know just
needs to test hico-det model, if you know, can you help me?
oh, thank you very much.