training error when using engine: CAFFE

[zjchuyp]

Hi, thanks a lot for your code. When I train, I get below errors, can you help me?
I0523 14:43:05.335593 4100 net.cpp:226] conv1/bn needs backward computation.
I0523 14:43:05.335593 4100 net.cpp:226] conv1 needs backward computation.
I0523 14:43:05.335593 4100 net.cpp:228] label_data_1_split does not need backwa
rd computation.
I0523 14:43:05.335593 4100 net.cpp:228] data does not need backward computation
.
I0523 14:43:05.335593 4100 net.cpp:270] This network produces output center_los
s_finetune
I0523 14:43:05.335593 4100 net.cpp:270] This network produces output softmax_lo
ss
I0523 14:43:05.335593 4100 net.cpp:283] Network initialization done.
I0523 14:43:05.335593 4100 solver.cpp:60] Solver scaffolding done.
I0523 14:43:05.335593 4100 caffe.cpp:252] Starting Optimization
I0523 14:43:05.335593 4100 solver.cpp:279] Solving FaceVggSoftmaxNet
I0523 14:43:05.335593 4100 solver.cpp:280] Learning Rate Policy: multistep
I0523 14:43:05.507194 4100 solver.cpp:337] Iteration 0, Testing net (#0)
I0523 14:46:06.717113 4100 solver.cpp:404] Test net output #0: center_loss_
finetune = 95.8247 (* 0.001 = 0.0958247 loss)
I0523 14:46:06.717113 4100 solver.cpp:404] Test net output #1: softmax_loss
= 12.702 (* 1 = 12.702 loss)
I0523 14:47:12.206027 4100 solver.cpp:228] Iteration 0, loss = 12.4081
I0523 14:47:12.206027 4100 solver.cpp:244] Train net output #0: center_loss
_finetune = 99.0989 (* 0.001 = 0.0990989 loss)
I0523 14:47:12.206027 4100 solver.cpp:244] Train net output #1: softmax_los
s = 12.309 (* 1 = 12.309 loss)
I0523 14:47:12.221627 4100 sgd_solver.cpp:106] Iteration 0, lr = 0.1
F0523 14:47:39.974076 4100 math_functions.cu:26] Check failed: status == CUBLAS
_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED
*** Check failure stack trace: ***

[shicai]

you may get some ideas from: BVLC/caffe#2417
or you may try trainning with smaller batch size.

[zjchuyp]

thanks a lot

[lrjun6188]

@zjchuyp I want to train the MobileNet on my own PC, but it always got a wrong, can you share your ideas or share your code?Really want to get your help