How to use momentum and weightdecay

[baggepinnen]

I have been trying to figure out how to use the weightdecay function and the Momentum optimizer.
It seems weightdecay should be passed to the optimiser function, e.g., when an optimizer is created. None of the optimizers do however have an option to pass in a weight decay parameter. Am I supposed to create a separate optimizer for weight decay and momentum that is called before the optimizer that calls descent?

[MikeInnes]

The optimiser APIs are pretty clumsy right now; they are supposed to be composable but it doesn't quite work. I'll take a look at redesigning this soon.

[baggepinnen]

I did manage to compose an optimizer with weight decay now when a Vector of optimizers is accepted by train!, but the code did not look pretty...

opt  = [ADAM(params(m), 0.01, decay=0.001); [weightdecay(Param(p), 0.002) for p in params(m) if isa(p, AbstractMatrix)]]

[MikeInnes]

Hopefully this is easier since #379.