[0.15] Out-of-memory with DetectNet

[lukeyeager]

Running DetectNet with the latest code on the caffe-0.15 branch runs out of memory. When I fall back to v0.15.2 I don't run out of memory. That means either #158 or #159 is almost certainly the culprit.

$ scripts/download_model_binary.py models/bvlc_googlenet/
$ export PYTHONPATH=`pwd`/python/
$ examples/kitti/train_detectnet.sh
...
I0616 13:29:27.153282 12145 caffe.cpp:231] Starting Optimization
I0616 13:29:27.153306 12145 solver.cpp:304] Solving
I0616 13:29:27.153311 12145 solver.cpp:305] Learning Rate Policy: step
I0616 13:29:27.168071 12145 solver.cpp:362] Iteration 0, Testing net (#0)
I0616 13:29:27.168102 12145 net.cpp:723] Ignoring source layer train_data
I0616 13:29:27.168108 12145 net.cpp:723] Ignoring source layer train_label
I0616 13:29:27.168112 12145 net.cpp:723] Ignoring source layer train_transform
F0616 13:29:28.951283 12145 gpu_memory.hpp:77] Check failed: try_reserve(size)
*** Check failure stack trace: ***
@     0x7fd56dfbd5cd  google::LogMessage::Fail()
@     0x7fd56dfbf433  google::LogMessage::SendToLog()
@     0x7fd56dfbd15b  google::LogMessage::Flush()
@     0x7fd56dfbfe1e  google::LogMessageFatal::~LogMessageFatal()
@     0x7fd56e615eb0  caffe::CuDNNConvolutionLayer<>::FindExConvAlgo()
@     0x7fd56e61a43e  caffe::CuDNNConvolutionLayer<>::Reshape()
@     0x7fd56e690868  caffe::Net<>::ForwardFromTo()
@     0x7fd56e690c17  caffe::Net<>::Forward()
@     0x7fd56e68492a  caffe::Solver<>::Test()
@     0x7fd56e68547e  caffe::Solver<>::TestAll()
@     0x7fd56e688299  caffe::Solver<>::Step()
@     0x7fd56e688fe9  caffe::Solver<>::Solve()
@           0x40c7ef  train()
@           0x4086b8  main
@     0x7fd56ca16830  __libc_start_main
@           0x408e29  _start
@              (nil)  (unknown)

/cc @drnikolaev @pooyadavoodi
I'll give you the dataset if you don't have it.

[lukeyeager]

Images WxHxC: 1248x384x3
Labels WxHxC: 16x25x1

[pooyadavoodi]

#173 solves the problem.