Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to implement the unbounded/bounded memory for while_op #7011

Closed
lcy-seso opened this issue Dec 25, 2017 · 1 comment
Closed

How to implement the unbounded/bounded memory for while_op #7011

lcy-seso opened this issue Dec 25, 2017 · 1 comment
Assignees
Labels

Comments

@lcy-seso
Copy link
Contributor

lcy-seso commented Dec 25, 2017

The concept of unbounded/bouned memory is fristed proposed by NTM . They can be used as a kind of external memory for complicated sequence tasks.

In attention mechanism, the encoder outputs can be regarded as an unbounded memory for the decoder. It means at each time step of decoder, the data in unbounded memory always remain constant, and the decoder read it. BUT, because an unbounded memory is involved in decoder's forward computation, it will also backpropagate the gradients it receives.

The unbounded/bounded memory is important for attention, but I do not find any example to show how to implement it?

@lcy-seso lcy-seso changed the title How to implement unbounded/bounded memory for while_op How to implement the unbounded/bounded memory for while_op Dec 25, 2017
@lcy-seso lcy-seso added the NMT label Jan 3, 2018
@lcy-seso
Copy link
Contributor Author

fix by this PR #7434

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants