WeightDrop in evaluation mode

[svengiegerich]

Hej Katerina, first thanks for this nice adaption! Helped me a lot to use the AWD-LSTM for my own code.
Not sure if you are still using this repo, but for everybody how comes along maybe this is helpful:

For DropConnect set_weights in WeightDropout.py faces an issue in newer (?) Pytorch versions, because torch.nn.functional.dropout returns a tensor during training but a parameter during evaluation. Adding (*) should fix the bug.

if not self.training: # (*)
    w = w.data

setattr(self.module, name_w, w)

For more infos, see salesforce/awd-lstm-lm#79.

[mourga]

Hey @svengiegerich thanks for the suggestion! I don't use the code anymore, so it'll be useful for people that work on this!