Provide an Adam solver

[slavnyk]

Are there any plans to add Adam solver to Caffe in the near future?
Adam: A Method for Stochastic Optimization
Python implementation (Theano)
Lua implementation (Torch)
I have been using the Lua implemenation for training neural networks for low-level image processing (such as noise reduction and super-resolution) with great success, but could not reproduce results of similar quality in Caffe with SGD/Adagrad/Nesterov. There are reports of better results (faster, more accurate) when training various types of deep networks using Adam.

[kashif]

I would like to try to do this, if no one else has started. I took the time to update the adam implementation for Lasagne and torch to the latest version of the paper. So next I will try to implement it in caffe.

[PatWie]

Just started to write the solver
https://github.com/PatWie/caffe/blob/adam/src/caffe/solver.cpp

Not sure if we really want to use

this->iter

for computing the effective stepsize.

[kashif]

Ok I was thinking perhaps:

int t = this->iter_ / this->param_.stepsize() + 1;

[shelhamer]

Thanks for the suggested solver. Closing as addressed by #2856 -- see the PR for status.