design of RNNOp #3727

Superjomn · 2017-08-28T18:15:54Z

wangkuiyi · 2017-08-28T19:13:00Z

doc/design/ops/rnn.md

+- init_memory, the variable to help initialize memory
+
+### step scopes
+Each RNN has more than one step times, and the stepnet will be executed in every step time.


The RNN might run one or more steps.

wangkuiyi · 2017-09-02T22:00:05Z

doc/design/ops/rnn.md

+
+<p aligh="center">
+<img src="./images/rnn.png"/><br/>
+fig 2 the RNN's data flow


fig 2 ==> Figure 2

wangkuiyi · 2017-09-02T22:00:26Z

doc/design/ops/rnn.md

+
+There are several important concepts:
+
+- stepnet, the network execute in every time step 


stepnet => step-net

wangkuiyi · 2017-09-02T22:00:50Z

doc/design/ops/rnn.md

+
+There are several important concepts:
+
+- stepnet, the network execute in every time step 


the network to be executed in each step

wangkuiyi · 2017-09-02T22:01:09Z

doc/design/ops/rnn.md

+- init-memory, the variable to help initialize state in the first time step.
+
+### step scopes
+


wangkuiyi · 2017-09-02T22:02:56Z

doc/design/ops/rnn.md

+- init-memory, the variable to help initialize state in the first time step.
+
+### step scopes
+


The step-net could have local variables defined. In each step of RNN execution, a scope is created to hold corresponding variables. Such a scope is known as a step scope.

wangkuiyi · 2017-09-02T22:18:58Z

doc/design/ops/rnn.md

+h_t = U h_{t-1} + W x_t
+$$
+
+Here, $h_t$ is time $t$'s state, $h_t$ is time $t-1$'s state, in implementation, we call the a variable that store a state memory.


", in implementation, we call the a variable that store a state memory." can be deleted

wangkuiyi · 2017-09-02T22:19:20Z

doc/design/ops/rnn.md

+$$
+
+Here, $h_t$ is time $t$'s state, $h_t$ is time $t-1$'s state, in implementation, we call the a variable that store a state memory.
+In step time $t$, $h_t$ is memory, $h_{t-1}$ is pre-memory (short for previous memory).


"In step time $t$, $h_t$ is memory, $h_{t-1}$ is pre-memory (short for previous memory)." can be deleted

wangkuiyi · 2017-09-02T22:21:24Z

doc/design/ops/rnn.md

+Here, $h_t$ is time $t$'s state, $h_t$ is time $t-1$'s state, in implementation, we call the a variable that store a state memory.
+In step time $t$, $h_t$ is memory, $h_{t-1}$ is pre-memory (short for previous memory).
+
+In each step scope


+In each step scope
+- each memory variable has a corresponding pre-memory variable
+- before a time step executes, copy (or make a reference) the value of previous step scope's memory to the pre-memory variable in current step scope.

=>

In the implementation, we can make an ex-memory variable either "refers to" the memory variable of the previous step, or copy the value of the previous memory variable to the current ex-memory variable.

wangkuiyi · 2017-09-02T22:22:03Z

doc/design/ops/rnn.md

+- each memory variable has a corresponding pre-memory variable
+- before a time step executes, copy (or make a reference) the value of previous step scope's memory to the pre-memory variable in current step scope.
+
+### C++ API


The C++ API

wangkuiyi · 2017-09-02T22:22:26Z

doc/design/ops/rnn.md

+- void Run(const framework::Scope& scope, const platform::DeviceContext& dev_ctx) const;
+ - run all the time steps.
+
+### User interface


The Python Interface

English grammar correction of rnn.md

wangkuiyi · 2017-09-11T19:39:17Z

doc/design/ops/rnn.md

+
+rnn = pd.create_rnn_op(output_num=1)
+with rnn.stepnet():
+ x = rnn.add_input(X)


This example uses rnn.add_input. But the next example uses rnn.segment_input. Are they the same?

yes, I will change all to rnn.add_input

need to differentiate two types of input: sequence input and static input. Each instance has different static input. But for one instance, it's same across all time steps.

the static inputs will be treated as global variables and that doesn't need to be passed as input.

the add_input statement only mark the sequence input that needs to be segmented for RNN's step times. @emailweixu

static inputs are different from parameters. They will still need to be splitted according to whether that instance is participating at a timestep, where parameters do not need to be splitted.

helinwang · 2017-09-11T20:47:42Z

doc/design/ops/rnn.md

+
+We can define an RNN's step-net using Block:
+
+```python


Does this API works with attention model?

This syntax should be compatible with Paddle V1, but without the support of Beam Search.

helinwang · 2017-09-11T20:49:29Z

doc/design/ops/rnn.md

+ # update current memory
+ h.update(new_state)
+ # indicate that h variables in all step scopes should be merged
+ rnn.set_output(0, h)


What does "0" mean in set_output? Every set_output in this PR uses "0" as argument.

0 means the "0-th" argument

wangkuiyi · 2017-09-13T22:48:12Z

doc/design/ops/rnn.md

+ h.update(
+ pd.matmul(W, sentence) + pd.matmul(U, h.pre_state()))
+ # get the last state as sentence's info
+ rnn.set_output(0, h)


Is the 0 here indicating the first output?

How can we specify that an RNN should return just the output from the last step?

rnn = pd.create_rnn_op() with rnn.stepnet(): x = rnn.set_inputs(X) # declare a memory (rnn's step) h = rnn.add_memory(init=a) # h.pre_state() means previous memory of rnn new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state())) # update current memory h.update(new_state) # indicate that h variables in all step scopes should be merged rnn.set_outputs(h) # output last step out = rnn(output_all_steps=False)

can we use the argument output_all_steps to output all steps or just the last step?

Superjomn and others added 3 commits August 28, 2017 14:14

add rnn

62fe33f

Update rnn.md

0470c9a

Update rnn.md

ea11172

wangkuiyi reviewed Aug 28, 2017

View reviewed changes

Superjomn and others added 6 commits August 28, 2017 12:17

Update rnn.md

9e9bc18

add image

ecb051a

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

636409f

Merge branch 'rnn_design' of github.com:Superjom/Paddle into rnn_design

fa5ddc2

add image

aaf20e9

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

30f992d

Superjomn requested review from jacquesqiao and zchen0211 August 29, 2017 01:43

Superjomn added 2 commits August 29, 2017 15:21

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

1a48300

add more details

6e6b978

lcy-seso changed the title ~~design of RNN for fix-length-setence~~ design of RNN for fix-length-sentence Sep 1, 2017

wangkuiyi reviewed Sep 2, 2017

View reviewed changes

Superjomn added 3 commits September 2, 2017 15:25

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

166b62a

fix grammer

27bb84d

update title

2737cfe

Superjomn changed the title ~~design of RNN for fix-length-sentence~~ design of RNNOp Sep 2, 2017

Superjomn added 7 commits September 2, 2017 17:02

refactor rnn user sample

1ff2295

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

3caf7bd

update rnn

28e2959

update image

8551acd

add more details

1f54435

finish

d065f04

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

cab04d4

Superjomn requested review from luotao1 and qingqing01 September 6, 2017 21:43

Update rnn.md

67bb3bf

Yi Wang and others added 5 commits September 11, 2017 11:55

Update

3f70bb7

Merge pull request #2 from wangkuiyi/rnn_design_yi

745f639

English grammar correction of rnn.md

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

7f79f52

add 2_level_rnn.dot

a1ab395

add rnn_2level_data.dot

71954ec

wangkuiyi reviewed Sep 11, 2017

View reviewed changes

segment_input -> add_input

270db83

helinwang reviewed Sep 11, 2017

View reviewed changes

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

4e46911

wangkuiyi reviewed Sep 13, 2017

View reviewed changes

Superjomn added 2 commits September 13, 2017 19:32

support output the final step

d48fbc3

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into rnn_design

1b952b1

wangkuiyi approved these changes Sep 13, 2017

View reviewed changes

Superjomn merged commit b3f6b5a into PaddlePaddle:develop Sep 14, 2017

Superjomn deleted the rnn_design branch September 14, 2017 01:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

design of RNNOp #3727

design of RNNOp #3727

Superjomn commented Aug 28, 2017 •

edited by wangkuiyi

Loading

wangkuiyi Aug 28, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 2, 2017

wangkuiyi Sep 11, 2017

Superjomn Sep 11, 2017

Superjomn Sep 11, 2017

emailweixu Sep 14, 2017

Superjomn Sep 15, 2017

emailweixu Sep 15, 2017

helinwang Sep 11, 2017

Superjomn Sep 11, 2017

helinwang Sep 11, 2017

Superjomn Sep 11, 2017

wangkuiyi Sep 13, 2017

Superjomn Sep 13, 2017


		There are several important concepts:

		- stepnet, the network execute in every time step

		- init-memory, the variable to help initialize state in the first time step.

		### step scopes

design of RNNOp #3727

design of RNNOp #3727

Conversation

Superjomn commented Aug 28, 2017 • edited by wangkuiyi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn commented Aug 28, 2017 •

edited by wangkuiyi

Loading