Simplify training interface by removing weight decay and scaling

[neubig]

There is some confusion regarding the interface of the "Trainer" class:
#641
#684

I agree that it's difficult to understand. This commit removes the rate decay and gradient scaling functionality that implicitly changes the learning rate in non-transparent ways. Here is an example of the before/after behavior:

Rate Decay Before

// At beginning of training
Trainer trainer(initial_learning_rate, rate_decay)
// After every epoch
trainer.update_epoch()

Rate Decay After

// At beginning of training
Trainer trainer(initial_learning_rate)
// After every epoch
trainer.learning_rate /= (1 - rate_decay)

Gradient Scaling Before:

cg.backward(loss)
trainer.update(scaling_factor)

Gradient Scaling After:

cg.backward(loss * scaling_factor)
trainer.update()

[pmichel31415]

LGTM!

[liesun1994]

I am wondering if lamtram need to be modified with the issues pulled.

[liesun1994]

When I am running lamtram， it may occurs Trainer::update_epoch has been deprecated and doesn't do anything. Please remove it from your code, and control the learning rate of the trainer directly, for example by: 'trainer.learning_rate /= (1 - rate_decay)', see #695 for details .

[yuvalpinter]

I can't seem to be able to run the trainer.learning_rate /= (1 - rate_decay) code. It gives me the following error: TypeError: 'property' object is not callable.
This happens for both MomentumSGDTrainer and AdamTrainer.

[shuheik]

Shouldn't it be
trainer.learning_rate *= (1 - rate_decay) or
trainer.learning_rate /= (1 + rate_decay)
instead of
trainer.learning_rate /= (1 - rate_decay)?

I assume rate_decay to be some small positive value and dividing by (1- rate_decay) would make learning_rate grow larger.