How to noise the LSTM?

[ethancaballero]

What method do you think would be best for noising the LSTM?
End of Noisy Network paper seems to suggest that randomization technique from "Bayesian recurrent neural networks" https://arxiv.org/abs/1704.02798 can be applied in order to noise the LSTM.

There're 2 TF implementations of Bayesian RNN:
https://github.com/DeNeutoy/bayesian-rnn/blob/master/bayesian_rnn.py
https://gist.github.com/mirceamironenco/06078722d729b968b9ab054744e136bc

[Kaixhin]

No idea. I think that if the noise is consistent over a trajectory then it should be OK, and frankly just having the final layers be noisy seems to be pretty good (if large portions of the network were noisy, I wonder how bad an influence this would be). I'll leave this issue open for others to discuss, but I'm not planning to investigate any further for now.

[ethancaballero]

Did you try noising the the embedding layer(s) (e.g. fc1)? Did it have positive/negative effect?

[Kaixhin]

Nope, feel free to try and see. In the paper they say:

When replacing the linear layers in the value and policy heads by noisy layers...

I'm not sure whether this means that they only use it in the output layers or not. I felt the last layer should be enough, but perhaps not (and probably this will depend on the problem).

[ethancaballero]

Also, scaling the noise in proportion to variance it causes in outputs like in OpenAI version might help as well:
https://arxiv.org/abs/1706.01905

^look at paragraph titled "Adaptive Noise Scaling" in Section 3

[Kaixhin]

Yep had a chat with some other people about this - both DM and OpenAI's contributions have their own pros and cons, and certainly it may be possible to combine them and get better results.

I'm making this repo more minimal to remove extra factors (such as GAE). For future reference, removing GAE maybe slightly decreases performance, but doesn't seem to have a massive impact on other metrics such as variance or stability.