Why the mAP is so low?

[chengshuai]

Hi, @sanghoon

I try train the example_train_384 in pvanet_obsolete(pva9.0) for pascal_voc 2007(trian:2007+2012,test:2007), the mAP is so low 13.1%，however, the mAP is 82.8% when i use you model. Why the result is different? Is there other change or trick during your training?

Thank you for your reply! The below is the script:

Training for 100k iterations

tools/train_net.py
--gpu 0
--solver models/pvanet_obsolete/example_train_384/solver.prototxt
--weights models/pvanet_obsolete/imagenet/original.model
--iters 100000
--cfg models/pvanet_obsolete/cfgs/train.yml
--imdb voc_2007_trainval
Testing

tools/test_net.py
--gpu 0
--def models/pvanet_obsolete/example_train_384/test.prototxt
--net output/faster_rcnn_pvanet/voc_2007_trainval/pvanet_frcnn_384_iter_100000.caffemodel
--cfg models/pvanet_obsolete/cfgs/submit_160715.yml

[yzhang123]

I have been wondering myself. I used the solver in models/pvanet/example_train/solver.txt
but is the difference between pvanet and pvanet_obsolete, and which should you use?

[chengshuai]

@yzhang123
i use the pvanet_obsolete slover.txt, the mAP is very low. Do you have the same result?

[dereyly]

Hi @chengshuai
My testing stats pascal_voc 2007(trian:2007+2012,test:2007):
pvanet_obsolete(pva9.0)
100k
MeanAP=0.7190

PVA 9.1
100k
Mean AP = 0.7512
120k
Mean AP = 0.7768
360k
Mean AP = 0.7922

ResNet50
80k
Mean AP = 0.7901 -- but slower at train and test

But I dont know why u have only 13.1% your test seems correct

[yzhang123]

@chengshuai after running
python tools/train_net.py --gpu 0 --solver models/pvanet_obsolete/example_train_384/solver_voc.prototxt --weights models/pvanet/pretrained/pva9.1_pretrained_no_fc6.caffemodel --iters 100000 --cfg models/pvanet_obsolete/cfgs/train.yml --imdb voc_2007_trainval+voc_2012_trainval

I get the error:
I0601 20:53:51.431869 10026 solver.cpp:60] Solver scaffolding done.
Loading pretrained model weights from models/pvanet/pretrained/pva9.1_pretrained_no_fc6.caffemodel
F0601 20:53:52.266515 10026 net.cpp:767] Check failed: target_blobs.size() == source_layer.blobs_size() (1 vs. 2) Incompatible number of blobs for layer conv2_1/1/conv
*** Check failure stack trace: ***
Aborted (core dumped)

Do you know this error? where did you get models/pvanet_obsolete/imagenet/original.model from?

[chengshuai]

Hi @yzhang123
The modles are downloaded:

1. Download full/original.model and move it to ./models/pvanet/full/
   2. Download comp/original.model and move it to ./models/pvanet/comp/

[chengshuai]

Hi @dereyly

My test result is:
pvanet_obsolete(pva9.0)
100k
MeanAP=0.131

PVA 9.1
100k
Mean AP = 0.731(smaller than your result Mean AP = 0.7512)

I do not know why the pva9.0 mAP is so low. I use a GPU Titan X for trainning, i do do not use the cudnn in Makefile.config.

[dereyly]

@chengshuai
Main idea that you use wrong weights caffemodel (for pva 9.0). Need to load pretrained from imagenet with batchnorms and not compressed.
Smaller resault on pva 9.1 may have 2 reasons:

1. forgot 2007+2012 (only 2007)
2. I think i have different solver. Standart solver is good, but I think it not tick till 100k iterasion with specific platoe step rule and in 100k iteration your learning rate is 0.001, when it tics to 0.0001 accuracy better. But i prefer 0.33 step

cudnn havent influense on acuuracy, but it strongly recomedeed for perfomance

[chengshuai]

@dereyly

Thanks, I will try again following your advice.

[bhushangawde]

Hi @sanghoon , @chengshuai ,

I tried the pvanet on KITTI dataset. I followed the steps given on the link rbgirshick#243.
But i am getting map as 0.09 for 100k iterations!.
When i plotted the loss i saw the loss is decreased only upto 1k iterations and it is constant afterwards
Please help!
Thankyou

[tzhang2014]

@chengshuai how much time use pvanet9.0 ?THX