Add constant parameter #9893
Add constant parameter #9893
Conversation
| self.const = self.params.get_constant('const', self.value) | ||
|
|
||
| def hybrid_forward(self, F, x, const): | ||
| return x + const |
There was a problem hiding this comment.
Should it be x + self.const?
There was a problem hiding this comment.
const is the parameter, and block handles it.
| @@ -548,6 +588,45 @@ def get(self, name, **kwargs): | |||
| setattr(param, k, v) | |||
| return param | |||
|
|
|||
| def get_constant(self, name, value=None): | |||
There was a problem hiding this comment.
When developing for a language model with weight drop, we took the approach of adding a new type of Parameter. While using the new WeightDropParameter, we found it quite difficult as user to properly change the type of parameter in the ParameterDict. Since dropout-on-parameter is a common use case too, should we be open to adding a getter function for such parameter as well? Or should we consider a different pattern?
There was a problem hiding this comment.
I think I've met some similar problems implementing the weight normalization, which also transforms the parameters.
There was a problem hiding this comment.
So the pattern here is that we need parameter transformation, and it can be done through overriding the data() (and maybe grad()) method. This currently requires inheriting the Parameter class, and there is no existing method in casting the parameter type. So in this case, what's needed is a way to cast the parameter.
One alternative is to provide a place to plug in transformation functions in data() and grad().
* add constant parameter * Update parameter.py * Update parameter.py * Update gluon.md
* add constant parameter * Update parameter.py * Update parameter.py * Update gluon.md
Description
(Brief description on what this PR is about)
Checklist
Essentials
make lint)Changes
Comments