Iteration 80, loss = nan, Training on Imagenet

[xiangyangli-cn]

I0508 12:39:32.293411 2763 net.cpp:166] Network initialization done.
I0508 12:39:32.293421 2763 net.cpp:167] Memory required for Data 210114408
I0508 12:39:32.293454 2763 solver.cpp:36] Solver scaffolding done.
I0508 12:39:32.293479 2763 solver.cpp:47] Solving CaffeNet
I0508 12:40:15.421900 2763 solver.cpp:208] Iteration 40, lr = 0.01
I0508 12:40:15.422210 2763 solver.cpp:65] Iteration 40, loss = 6.99261
I0508 12:40:46.141950 2763 solver.cpp:208] Iteration 80, lr = 0.01
I0508 12:40:46.142240 2763 solver.cpp:65] Iteration 80, loss = nan
I0508 12:41:18.104593 2763 solver.cpp:208] Iteration 120, lr = 0.01
I0508 12:41:18.104933 2763 solver.cpp:65] Iteration 120, loss = nan
I0508 12:41:49.053544 2763 solver.cpp:208] Iteration 160, lr = 0.01
I0508 12:41:49.053845 2763 solver.cpp:65] Iteration 160, loss = nan
what's is the problem?? Anyone can help me?

[tdomhan]

@jeffdonahue, I wonder if this is in any way related to #393. There's also an issue about getting a nan, however much earlier in the process and it looks like your loss is just taking off, to infinity. Did you try a smaller learning rate?

[shelhamer]

Closing as this couldn't be reproduced. If you are trying to fit training into a smaller device see #430.