Adding the interface for the momentum optimizer #4919
Conversation
| Args: | ||
| block: the block in which the loss variable is present | ||
| """ | ||
| pass |
There was a problem hiding this comment.
raise NotImplementedError()
There was a problem hiding this comment.
just keep using pass, because sometimes user did not need to impliment this method~
| pass | ||
| # Dictionary of accumulators. | ||
| # {accum_name : { paramter_name : accumulator_for_parameter, ...}, ...} | ||
| self._accumulators = defaultdict(lambda: dict()) |
There was a problem hiding this comment.
should add some comment about what is accumulator
There was a problem hiding this comment.
Fixed in latest commit
| """ | ||
| create and add gradient Operators in BlockDesc to Compute gradients of `loss` | ||
| for parameters in parameter_list | ||
| pass |
There was a problem hiding this comment.
raise NotImplementedError()
| class MomentumOptimizer(Optimizer): | ||
| """Simple Momentum optimizer with velocity state | ||
| """ | ||
| _velocity_acc_str = "velocity" |
There was a problem hiding this comment.
I am not familiar with this usage of Python, should we put this value into
def __init__():
self. _velocity_acc_str = "velocity"not sure which is better
There was a problem hiding this comment.
This way _velocity_str is a like a static member of the class MomentumOptimizer. However, on doing self. _velocity_acc_str = "velocity", it will become an instance member. Since this is a class-wide constant, I preferred to make it a static constant.
fixes #4898