the init parameter "optimizer" of Trainer() should be a function

[jacquesqiao]

Background

@seiriosPlus find a bug in the new Trainer high-level API when writing a model with learning rate decay. The optimizer is created in a different program than the trainer use.

Reason and solution

Paddle/python/paddle/fluid/trainer.py

Lines 88 to 103 in 9f0dcfd

	class Trainer(object):
	"""
	Args:
	train_func(callable): A function which will return loss. The loss must be a scalar.
	optimizer(optimizer.Optimizer): The optimizer should be an instance of Optimizer
	place: The device place of this trainer.
	"""
	def __init__(self,
	train_func,
	optimizer,
	param_path=None,
	place=None,
	parallel=False):
	self.__stop = False

In the design of Trainer API, train_program is a function call, it will be called inside Trainer under a with scope

Paddle/python/paddle/fluid/trainer.py

Lines 115 to 117 in 9f0dcfd

	with framework.program_guard(self.train_program, self.startup_program):
	program_func_outs = train_func()

But the init parameter optimizer is an object, it will be created outside the with scope, this is not right, so we should make optimizer also a function call that return an optimizer boject

the interface should be

class Trainer(object):
    def __init__(self,
                 train_func,
                 optimizer_func,
                 param_path=None,
                 place=None,
                 parallel=False):

[seiriosPlus]

when we init Optimizer with the code of Fluid, the variables will be created outside the scope.
For example:

optimizer = fluid.optimizer.Momentum(
                learning_rate=fluid.layers.piecewise_decay(
                    boundaries=bd, values=lr),
                momentum=0.9,
                regularization=fluid.regularizer.L2Decay(1e-4))

[jetfuel]

@jacquesqiao @seiriosPlus Does the bug only happen with the learning rate decay? I wonder if there is a way to create a copy of the optimizer with the proper.

[sidgoyal78]

@jacquesqiao : I think your suggestion sounds pretty good. I will make a PR implementing your approach.