Again: Training imagenet: loss does not decrease

[dereyly]

I have the same problem as #401
But have different story. Half year ago I sucsefully learned Imagenet caffe net with cuda 6.5 and cudnn_v2 (45000 iter with same resault as base model). And now i try to learn it again and failed -- loas doesn't decrease it is about 6.9 and accuracy =0. The sistem and driver is the same.
Test of old model is ok (expected accuracy).

I read the issues and there is no problem with shuffle the lmbd data is the same and same protobuff and build

Before write my question I rebuild new caffe with cuda 7.0 and cudnn_v3 and new driver 346.39 and have the same problem.

I run test about 10k-20k iterations. My old save have 16% acuracy on 10k iteratioons now its 0.

[baiyancheng20]

hi, have you resolved the problem? I met this problem, too.

[dereyly]

Hi. Yes. I use MSRA init in convolution layers and same gaussian in full connected
But later I try LSUV: https://github.com/ducha-aiki/LSUVinit (All you need is good init)
it is better then MSRA
Finally batch normalization may solve this problem too

[baiyancheng20]

@dereyly thanks you. I use the MSRA init, and it works. Have you re-trained the alex network? Did it achieve the same test accuracy (58% on val)?

[dereyly]

Yes. I train 450k (but u can stop earlier) iterations and get same result top1=58% and top5=81%