hi, what about the performance on small dataset like v-coco?

[bitwangdan]

i run the " python tools/Test_VCL_ResNet_VCOCO.py --num_iteration 200000"
the results are a little bit poor

[zhihou7]

VCL_union_multi_ml1_l05_t3_rew_aug5_3_new_VCOCO_test has only 39.31? It is wired. If you obtain around 47.8 with VCL and around 47.2 without VCL, I think the result is reasonable. 39.31 is wired. I'll check the code and test my model with the released code again.

[bitwangdan]

VCL_union_multi_ml1_l05_t3_rew_aug5_3_new_VCOCO_test has only 39.31? It is wired. If you obtain around 47.8 with VCL and around 47.2 without VCL, I think the result is reasonable. 39.31 is wired. I'll check the code and test my model with the released code again.

ok, looking forward to your reply

[zhihou7]

Here is the model parameters.

VCOCO: https://drive.google.com/file/d/1X8XZ7sycQ7GM1uvT6xVSRnNisw3QSnrt/view?usp=sharing. I test the result is 47.82. The model in my reported result is deleted by accident (the baseline also decreases).

HICO: https://drive.google.com/file/d/16unS3joUleoYlweX0iFxlU2cxG8csTQf/view?usp=sharing

[bitwangdan]

@zhihou7
Thank you for your reply，I have a few questions：
1.What is the base network on which you provide the model?
2.The 39.31 Mentioned above is the result of the model trained with "python tools/Train_VCL_ResNet_VCOCO.py --model VCL_union_multi_ml1_l05_t3_rew_aug5_3_new_VCOCO_test --num_iteration 400000", The dataset was also downloaded according to the address you provided,Maybe there are some bugs in the code，when I train the model, sometimes the loss is nan。
BTW，I did not experiment on the HICO dataset.

[zhihou7]

Yeah, I'm testing the code. The model is also based on the released code. But I trained it before I clear the code to release. I may remove some code by mistake. Empirically, the hyper-params l05 and re-weighting will improve our baseline to 47.0.

Thank you for pointing out the problems.

[bitwangdan]

@zhihou7 ok！Look forward to your results

[zhihou7]

emmm, my result seems normal. 20000 iters, 46.4.

[bitwangdan]

emmm, my result seems normal. 20000 iters, 46.4.

Well, I'll try again

[zhihou7]

You can simply test the model in iteration 10000.

[bitwangdan]

@zhihou7 hi, i tried again，my result is much worse than yours, 20000 iters

[zhihou7]

Hi, Did you test the model: https://drive.google.com/file/d/1X8XZ7sycQ7GM1uvT6xVSRnNisw3QSnrt/view?usp=sharing ? Is it also much worse than 47?

[bitwangdan]

@zhihou7 hi, I tested the model you provided again，260000 iters, 47.83%
but the model I trained based on your code has poor results，Is there any difference in this?

[zhihou7]

Ok, I guess you can obtain 46.4 with [this model] (https://drive.google.com/file/d/1X8XZ7sycQ7GM1uvT6xVSRnNisw3QSnrt/view?usp=sharing) . I trained this model 2 days ago on the released code. I'm testing the released code on a new machine and re-download the data.

Did you miss some data or something? It is really wired.

[bitwangdan]

@zhihou7 ok，I'll look again to see if I missed something。

[bitwangdan]

@zhihou7 hi，Do you have any results? Which backbone did you use for training?

[zhihou7]

I use resnet50. I haven't run the code. I have to queue for GPU...

[bitwangdan]

@zhihou7 I also use resnet50, I hope a third person can prove it。

[zhihou7]

Did you ever run iCAN or TIN on V-COCO? My code is fully based on their code besides the VCL.py file. I need to queue until tonight for a GPU in the new GPU machine.

[bitwangdan]

@zhihou7 oh, thanks, and i tried the ican model on V-COCO and the result is normal.

[zhihou7]

Ok, thanks. Then, that's more likely to be a problem with my released code. I will test it fully.

[zhihou7]

@bitwangdan what's your tensorflow version? The data that you used for VCL is same as iCAN? I download the released code and use the data that I downloaded before to test code. I still get normal result (45.69) in iteration 20000. In my current GPU server, I can not download the data from google drive. I will test it tonight.

[bitwangdan]

@zhihou7 hi,
tf: 1.14
python:3.6
cuda:10.1

[zhihou7]

Your environment is same as ours. I have finished my test on a fully new server as the steps:

git clone https://github.com/zhihou7/VCL.git
cd VCL
bash ./misc/download_dataset.sh 
bash ./misc/download_training_data.sh
python tools/Train_VCL_ResNet_VCOCO.py --model VCL_union_multi_ml1_l05_t3_rew_aug5_3_new_VCOCO_test --num_iteration 20000
python tools/Test_ResNet_VCOCO.py --num_iteration 20000

I obtain the result:

---------Reporting Role AP (%)------------------
           hold-obj: AP = 33.41 (#pos = 3608)
          sit-instr: AP = 21.79 (#pos = 1916)
         ride-instr: AP = 62.81 (#pos = 556)
           look-obj: AP = 18.21 (#pos = 3347)
          hit-instr: AP = 75.83 (#pos = 349)
            hit-obj: AP = 52.01 (#pos = 349)
            eat-obj: AP = 35.81 (#pos = 521)
          eat-instr: AP = 7.64 (#pos = 521)
         jump-instr: AP = 53.76 (#pos = 635)
          lay-instr: AP = 24.75 (#pos = 387)
talk_on_phone-instr: AP = 49.59 (#pos = 285)
          carry-obj: AP = 39.62 (#pos = 472)
          throw-obj: AP = 42.53 (#pos = 244)
          catch-obj: AP = 47.51 (#pos = 246)
          cut-instr: AP = 36.07 (#pos = 269)
            cut-obj: AP = 36.14 (#pos = 269)
 work_on_computer-instr: AP = 59.01 (#pos = 410)
          ski-instr: AP = 45.80 (#pos = 424)
         surf-instr: AP = 79.82 (#pos = 486)
   skateboard-instr: AP = 83.14 (#pos = 417)
        drink-instr: AP = 34.78 (#pos = 82)
           kick-obj: AP = 66.62 (#pos = 180)
        point-instr: AP = 0.03 (#pos = 31)
           read-obj: AP = 34.55 (#pos = 111)
    snowboard-instr: AP = 74.44 (#pos = 277)
Average Role [scenario_1] AP = 44.63
---------------------------------------------
Average Role [scenario_1] AP = 46.49, omitting the action "point"

I notice I changed the lr to 0.01 compared to iCAN. Do you use 0.01 or 0.001? I forget the effect of learning rate.

Here is the log file with the first 1000 iterations.

test.txt

[bitwangdan]

@zhihou7 hi，I renewed my environment，this is my result, 200000 iters，this looks like a normal result,thank you very much for your reply during this period, this is a very good job。