Adjusting optimizer learning rates on-the-fly

[staticfloat]

It would be great to allow the Flux optimizers to have parameters that can be tweaked from the outside. As an example, I would like to have an SGD where I can tune the learning rate over time. I could do this by destroying the optimizer and recreating it every epoch, (which is fine for SGD) but that hurts performance with optimizers that keep running estimates of gradient magnitude such as ADAM or RMSProp.

[tejank10]

I am thinking of treating the optimizers as structs rather than functions. For example, ADAM will be a struct having learning rate, momentum, velocity etc. as its fields. And there can be an update function which takes this struct and performs the update, thereby updating the momentum and velocity. In this way, we'll be able to mutate the learning rate.

[iblislin]

Currently, I tweak learning rate via opening the training loop and changing the Δ of back!(loss, Δ).

[MikeInnes]

Fixed by #379.