block design #3708

Superjomn · 2017-08-27T21:51:17Z

No description provided.

QiJune · 2017-08-28T03:35:21Z

doc/design/block.md

+
+Paddle will have `if_else_op`, `switch_op`, `while_op` and some other operators that need a function block such as RNNOp, and enable definition and execution of a block is vital to those operators.
+
+What's more, Paddle should has a `pd.namespace`, which will help to preventing name conflicts of variables in large model.


What's the relationship between namespace and scope?

namespace is on the top of a scope, namespace just add a namespace-prefix to the local variable in a scope.

If so, we only need one global scope, and we use different namespace to prevent name conflicts. Am I right?

Yes.

Scope is from Caffe2's WorkSpace, op developers is free to use Scope when implement new operators, but we will provide pd.namespace to users, and hide the Scope concept from them.

@QiJune To what I understand from this design doc draft, the namespace is one of the several uses of Block. Other uses include IfOp, WhileOp, RNNOp. @Superjom am I right?

In my understanding, namespace is more widely used then block, for example, we can define a function inside a namespace without block.

QiJune · 2017-08-28T03:40:44Z

doc/design/block.md

+
+with `pd.namespace`, user create 3 different namespaces and in each one, the parameters those have specific name will be reused.
+
+## Why use Block to replace NetOp


What's the lifecycle of a block?

currently, block is maintained by python, no harm I think

I think the lifecycle of a Block is a very good question and a must-to-answer question in this design doc. Because Block is a C++ class, its lifecycle should be discussed in the scope of the C++ core.

wangkuiyi · 2017-08-28T03:16:18Z

doc/design/block.md

@@ -0,0 +1,128 @@
+# Block design
+
+In computer programming, a block is a lexical structure of source code which is grouped together.


According to Wikipedia

a block or code block is a lexical structure of source code which is grouped together.

In C++ and Java, a block is represented by a pair of curly braces, and is used in control structures like if, for, and function definitions.

Blocks do not only group source code, but also

narrow the lexical scope of variables so that they do not conflict with variables having the same name used elsewhere in a program.

yes, this Block will be used by if_else_op, while_op and RNNOp, RNNOp will use a Block as a function.

I wrote this segment for your reference how to change the design doc. I was trying to rewrite this paragraph.

wangkuiyi · 2017-08-28T03:20:47Z

doc/design/block.md

+In Paddle, we need a similar concept to support following scenes:
+
+- the basic unit of execution, just like the original `NetOp`, block can group multiple operators and make them behave just like a single operator. Unlike `NetOp`, a network should not be an operator, but a block can be treated as a operator.
+  - Block should have a `Exec` method, which will execute the operators grouped in the block.


I think Block::Exec should also "realize" local variables, i.e., creating a Scope which is like a stack frame, and creating variables in it. I think so because it is Block who knows about the local variable definitions, thus it is the only entity who can realize these variables.

Currently, only one Scope is needed, and variables in Blocks are stored in this scope.

All the Variables will have unique names, so local variables in a Block will not affect the global namespace.

There is no difference whether a Block has a local scope or not.

@wangkuiyi the local variables inside a block will be used by the backward net too, so they should be able to be visited by Operator outside this block.

wangkuiyi · 2017-08-28T03:23:07Z

doc/design/block.md

+- Support the definition and execution of special operators such as `RNNOp`, `switch_op`, `while_op`, `if_else_op`, which needs one or more code-blocks.
+  - Block will help to define and execute thouse operators.
+
+## when need a block


when need a block 没有主语。这里应该是 When is Block needed 或者 How is Block used，这样 Block 才是主语.

wangkuiyi · 2017-08-28T03:23:28Z

doc/design/block.md

+  - Block will help to define and execute thouse operators.
+
+## when need a block
+Blocks are needed in following code snippets


Blocks are used in control structures and namespace definitions.

wangkuiyi · 2017-08-28T03:25:02Z

doc/design/block.md

+}
+```
+
+Paddle will have `if_else_op`, `switch_op`, `while_op` and some other operators that need a function block such as RNNOp, and enable definition and execution of a block is vital to those operators.


Similarly, we propose to use Block with IfOp, IfElseOp, SwitchOp, WhileOp, RNNOp.

wangkuiyi · 2017-08-28T03:26:30Z

doc/design/block.md

+
+Paddle will have `if_else_op`, `switch_op`, `while_op` and some other operators that need a function block such as RNNOp, and enable definition and execution of a block is vital to those operators.
+
+What's more, Paddle should has a `pd.namespace`, which will help to preventing name conflicts of variables in large model.


We'd also like to introduce a namespace API for PaddlePaddle, which likes the C++ namespace, names a Block and prefixes the names of variables defined in it.

wangkuiyi · 2017-08-28T03:28:07Z

doc/design/block.md

+
+with `pd.namespace`, user create 3 different namespaces and in each one, the parameters those have specific name will be reused.
+
+## Why use Block to replace NetOp


I think Block is a C++ class derived from OperatorBase, am I correct?

I'd thought Block is basically a rename of NetOp, with Block::Run creates a new scope for local variable realization, am I correct?

wangkuiyi · 2017-08-28T03:34:04Z

doc/design/block.md

+```
+
+```python
+rnn = pd.rnn_op()


As we analogy the DL system with programming languages in this design, I am thinking that the step net to the RNNOp is like a C++ function step_net to a caller function RNNOp:

void RNNOp( const vector<Something>& input, vector<Something>* output, function<Something(Something, int*)> step_net) { for (const auto& in_element : input) { output->push_back(step_net(in_element, 0/*initial memroy value*/)); } } Something step_net(const Something& input_element) { return DoSomething(input_element) }

However, this implementation doesnt' handle the "memory". In C++, it is true that we can add a memory like:

void RNNOp( const vector<Something>& input, vector<Something>* output, function<Something(Something, int*)> step_net) { Something memory = 0; for (const auto& in_element : input) { output->push_back(step_net(in_element, 0/*initial memroy value*/)); } } Something step_net(const Something& input_element) { memory = Update(input_element, memory); return DoSomething(input_element, memory); }

However, above solution doesn't work in a DL system, because each step needs to have its local copy of memory, so could all these memories be kept for being used by the backward algorithm later. An intuitive solution is to define all these memories in RNNOp:

void RNNOp( const vector<Something>& input, vector<Something>* output, function<Something(Something, int*)> step_net) { vector<MemoryType> memories; memories.resize(input.size()+1); memories[0] = kInitialMemoryValue; for (int i = 1; i <= input.size(); i++) { output->push_back(step_net(input, memories, i)); } } Something step_net( const vector<Something>& input, vector<MemoryType>* memories, int step) { memories[step] = Update(memories[step-1], input[step]); return DoSomething(input[step], memories[step]); }

Some might challenge that RNNOp doesn't know MemoryType at all -- only step_net knows the type. However, in our design, or, in any dynamic-typed programming languages, both MemoryType and Something are actually Variable. So it seems that this approach works.

But anyway, it still has a problem -- Block doesn't implement functions, but only the body of a function definition. To mimic the invocation of step_net in RNNOp, we need either creates a class Function derived from Block, or we just don't use function calls in above example -- instead, we can use just C++ blocks:

void RNNOp( const vector<Something>& input, vector<Something>* output) { vector<MemoryType> memories; memories.push_back(kInitialMemoryValue); int step; void step_block() { # We use a C++ function without parameter nor return value to mimic block memories.push_back(Update(memories[step-1], input[step])); output->push_back(DoSomething(input[step], memories[step])); } for (int step = 1; step <= input.size(); step++) step_block(); }

Please be aware that the sub-scope created by parnet_scope->NewScope() has a parent pointer and can be used to access all variables defined in parent_scope, so our Block mechanism is like defining a void function (a function without parameter nor return value) inside the C++ RNNOp function, so it can access variables defined in RNNOp.

Currently, the RNNOp is a rough implementation, and embedded all the details inside operator such as output->push_back.

We will have a RNNStep operator, which will expose most details as interfaces to python, and maybe in that implementation, some details may be same as this comment.

…ck_design.md

Superjomn · 2017-08-30T18:43:47Z

doc/design/block.md

+  Block(Block* father, Scope* scope) : father_block_(father), scope_(scope) {}
+
+  // NOTE Block has its own scope, so the argument scope here is not needed.
+  virtual void Run(const Scope& scope,


If Block is an operator, the Run interface will always get a scope, what's the relationship between this scope and Block's own scope that is created or passed in by the constructor?
@wangkuiyi @jacquesqiao

wangkuiyi · 2017-09-02T21:23:29Z

doc/design/block.md

+struct BlockDesc;
+
+// VarDescScope is similar to Scope, but offer a simpler map with the same value type.
+class VarDescScope {


I don' think we need to add new class VarDescScope. All members here can be moved into class Block.

wangkuiyi · 2017-09-02T21:24:22Z

doc/design/block.md

+  // lookup a variable description, recursively or not.
+  VarDesc* Lookup(const std::string& name, bool recursive=true);
+  // create a new description in local namescope.
+  VarDesc* New(const std::string& name);


Let's don't have New and Delete here, because a Block is constructed from a BlockDesc proto message, and there shouldn't be a change for users to call New and Delete to add/delete variable definitions in a Block.

wangkuiyi · 2017-09-02T21:34:05Z

doc/design/block.md

@@ -0,0 +1,238 @@
+# Block design


# Design Doc: Use Block in RNNOp, WhileOp, IfElseOp

wangkuiyi · 2017-09-02T21:36:38Z

doc/design/block.md

@@ -0,0 +1,238 @@
+# Block design
+
+In C++ and Java, a block is a lexical structure of source code which is grouped together.


RNNOp looks like the loop structure in programming languages. And similarly, WhileOp and IfElseOp are like loop and conditions respectively. So we want to verify if we should have a class Block in PaddlePaddle that works like a pair of curly braces in the loop and condition structures of programming languages.

wangkuiyi · 2017-09-02T21:37:41Z

doc/design/block.md

+# Block design
+
+In C++ and Java, a block is a lexical structure of source code which is grouped together.
+


Let's start from how RNN is described using PaddlePaddle:

v = some_op() m_boot = some_op() W = pd.Variable(shape=[20, 20]) U = pd.Variable(shape=[20, 20]) rnn0 = RNNOp() with rnn0.stepnet() as net: # declare stepnet's inputs x = net.add_input(v) # declare memories h = net.add_memory(m_boot) fc_out = pd.matmul(W, x) hidden_out = pd.matmul(U, h) sum = pd.add_two(fc_out, hidden_out) act = pd.sigmoid(sum) # declare stepnet's outputs net.add_output(act, hidden_out) acts, hs = rnn0()

…ck_design.md

wangkuiyi · 2017-09-04T18:04:14Z

doc/design/block.md

+W = pd.Variable(shape=[20, 20])
+U = pd.Varable(shape=[20, 20])
+
+rnn = RNNOp()


Consulting https://github.com/PaddlePaddle/Paddle/pull/3828/files, should this line be something likernn = create_rnn_op(inputs = [...], output_num=...)?

wangkuiyi · 2017-09-04T18:05:49Z

doc/design/block.md

+rnn = RNNOp()
+with rnn.stepnet() as net:
+  x = net.add_input(v)
+  h = net.add_memory(init=m_boot)


Do we need some explanation about how does add_memory works?

wangkuiyi · 2017-09-04T18:07:01Z

doc/design/block.md

+acts, hs = rnn()
+```
+
+This python program will be transformed into Protobuf messages which describe the model, a C++ Block class will create the corresponding Variables and Operators, and execute all the operators.


This python program will be transformed into Protobuf message

==>

Above Python program builds a protobuf message

wangkuiyi · 2017-09-04T18:08:14Z

doc/design/block.md

+a = pd.Varaible(shape=[20, 20])
+b = pd.fc(a, params=["fc.w", "fc.b"])
+
+rnn = pd.create_RNNOp()


It seems that pd.create_RNNOp here doesn't match above invocation to RNNOp.

wangkuiyi · 2017-09-04T18:08:48Z

doc/design/block.md

+```c++
+class Block {
+public:
+  Block(Block* father) : father_block_{father} {}


father ==> parent

wangkuiyi · 2017-09-04T18:09:09Z

doc/design/block.md

+  Block(Block* father) : father_block_{father} {}
+  // add a new VarDesc, return the pointer to enable other functions.
+  // NOTE Will check whether some variable called the same name.
+  VarDesc* AddVarDesc();


AddVarDesc => CreateVar?
AddOpDesc => CreateOp?

AddVarDesc and AddOpDesc registers Protobufs of Variable and Operators.

But CreateVar means create the Variables.

wangkuiyi · 2017-09-04T18:09:39Z

doc/design/block.md

+  VarDesc* FindVarDesc(const string& name, bool recursive=true);
+
+private:
+  Block* father_block_;


father_block_ => parent_

wangkuiyi · 2017-09-04T18:09:56Z

doc/design/block.md

+private:
+  Block* father_block_;
+  // descriptions
+  map<VarDesc> var_descs_;


var_descs_ => vars_
op_descs => ops_

this example is a code snippet of defining the Protobuf in Block, var_descs_ and op_descs_ store description.

ops_ member will be used to store OperatorBase*, and vars_ member will be used to store Variable*.

wangkuiyi · 2017-09-04T18:10:48Z

doc/design/block.md

+  }
+
+  // create all variables and operators according to Protobuf description.
+  RuntimeInit() {


RuntimeInit => InitRuntime ?
LocalScopeInit => CreateLocalScope ?

wangkuiyi · 2017-09-04T18:12:01Z

doc/design/block.md

+  }
+
+protected:
+  void LocalScopeInit(const framework::Scope &scope) {


This function has only a few lines, and it only called by Run, should we move these a few lines into Run?

…esign.md

putcn · 2017-09-11T18:11:17Z

doc/design/block.md

+acts, hs = rnn()
+```
+
+Above Python program builds a protobuf message which describe the model, a C++ Block class will create the corresponding Variables and Operators, and execute all the operators.


it would be great if we can show the protobuf created from above code, so that reader can better connect python coding block and the concept Block in paddle

helinwang · 2017-09-11T18:13:11Z

doc/design/block.md

+  // be proposed and embedded into pybind to enable python operate on C++ pointers.
+  VarDesc* FindVar(const string& name, bool recursive=true);
+
+  OpDesc* FindOp(const string& name);


Why FindOp does not have recursive parameter? (it looks very similar to FindVar, and FindVar has the parameter)

Variables in parent block may be referenced by child block, but op won't.

So variables need a recursive Find, but op doesn't need.

helinwang · 2017-09-11T18:15:02Z

doc/design/block.md

+  OpDesc* FindOp(const string& name);
+
+  BlockDesc Serialize() const;
+  void DeSerialize();


Why DeSerialize does not take any parameter? I thought it should deserialize from something.

yes, my mistake.

putcn · 2017-09-11T18:17:38Z

doc/design/block.md

+
+During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).
+
+VarDesc in a block should have its name scope to avoid local variables affect father block's name scope. 


"should have its name scope" is confusing, may this be "should have its scope named"?

helinwang · 2017-09-11T18:18:20Z

doc/design/block.md

+
+class Block : OperatorBase {
+public:
+  // desc may be serialized from a RuntimeTable or not.


Where is the method that creates a BlockDesc from RuntimeTable?

RuntimeTable will be merged into Block as its Construction, it takes a BlockDesc as input, and creates operator instances for Block.

putcn · 2017-09-11T18:18:35Z

doc/design/block.md

+
+During the generation of the Protobuf message, the Block should store VarDesc (the Protobuf message which describes Variable) and OpDesc (the Protobuf message which describes Operator).
+
+VarDesc in a block should have its name scope to avoid local variables affect father block's name scope. 


affect-> affecting

helinwang · 2017-09-11T18:22:05Z

doc/design/block.md

+  // desc may be serialized from a RuntimeTable or not.
+  Block(const BlockDesc& desc) desc_(desc) {}
+
+  void InferShape(const framework::Scope& scope) const override {


What does InferShape do exactly? Does it just allocate temporary variables?

helinwang · 2017-09-11T18:23:21Z

doc/design/block.md

+  }
+
+  void Run(const framework::Scope& scope,
+           const platform::DeviceContext& dev_ctx) const override {


What happens if ops from a block require to run on different device contexts? (e.g., one OP can run on CPU only, other OPs must run on GPU).

this is a historical issue, Block inherent from OperatorBase, so the same interface of Run.

I think multi-device is a very important feature and we should take this into consideration at current desgin.
The basic elements of Block(or Graph) is actually Nodes and Edges. Node has a device attribute to decide which device to run. Device includes CPU/MKL/CUDA/FPGA/etc. Some developers from inf will contribute FPGA code for refactoring Paddle. And FPGA is not suitable for all operators, and will only optimize for specific operators. So, multi-device(CPU/FPGA) is needed.
Please refer to #3943

And, What's the relationship between these two concepts, Block and Graph? If Block is Graph, and the private member of Block should be Nodes and Edges. The Graph mainly describes data dependency and control dependency. And Graph will run by an Executor, the executor will support multi-device executing.

better to discuss this in another issue.

putcn · 2017-09-11T18:25:30Z

doc/design/block.md

+
+out = rnn()
+```
+the method `pd.get_variable` can help retrieve a Variable by a name, a Variable may store in a father block, but might be retrieved in a child block, so block should have a variable scope that supports inheritance.


the method pd.get_variable retrieves Variable by name. a Variable may be stored in the current block or any ancestor block, that's why Block needs to support scope inheritance.

frankly, I'm bit lost when switching between concepts of Block and Scope. looks Scopes' inheritance chain are connected by the hierarchy of Blocks? Do I understand it correctly? We may need to have a clearer statement on these 2 concepts.
also, from above example, I don't quite see the inheritance, I can only see that a var named "fc.w" is created and get referenced by that name in later code.

helinwang · 2017-09-11T18:27:22Z

doc/design/block.md

+    }
+  }
+
+  void Run(const framework::Scope& scope,


To harness multiple CPU cores, we need a scheduler to run the OPs (none interdependent OPs should be able to be scheduled concurrently on different thread in a thread pool).

However, making Run a method of Block, we are limiting ourselves to run the OPs sequentially. Feels like we should have a scheduler who runs the block (or graph), in my opinion Run should not be a method of Block. Block should be just a description of what to run, rather than how to run it.

we may add a DependencyEngine latter, that will make block paralelly run.

…esign.md

helinwang · 2017-09-11T18:31:06Z

doc/design/block.md

+Block inherits from OperatorBase, which has a Run method. 
+Block's Run method will run its operators sequentially.
+
+There is another important interface called `Eval`, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block, 


Feels like we need a function maybe called Prune which generate a subgraph (or maybe block) from a graph. And then call Run to run it.

good idea, I will add a function called Prune

putcn · 2017-09-11T18:47:50Z

doc/design/block.md

+After all the description of variables and operators is added into SymbolTable,
+the block has enough information to run.
+
+To make block simpler, we use another class `RuntimeTable` to create Variable and Operator objects. `RuntimeTable` is free to add memory optimizements in the future.


RuntimeTable is free to add memory optimizements in the future. -> RuntimeTable will add memory optimizations in the future.

do I understand this statement correctly?

putcn · 2017-09-11T18:55:52Z

doc/design/block.md

+Block inherits from OperatorBase, which has a Run method. 
+Block's Run method will run its operators sequentially.
+
+There is another important interface called `Eval`, which take some arguments called targets, and generate a minimal graph which takes targets as the end points and creates a new Block, 


we may want to mention why Eval is needed and in what case it is used.

helinwang · 2017-09-11T19:01:27Z

In my opinion this block design has two issues:

Block Run implies sequential run, which does not harness multiple CPU cores.
Block Run takes a single platform::DeviceContext as argument, which means a block in which some OPs can only run on CPU and some OPs can only run on GPU, can not run.

These two issues seems to come from the fact that Block is a subclass of OperatorBase. From my understanding, Block is more like something that describe the dependency of Operators. Perhaps we only need Operator to be able to Run, and our runtime only runs Operator, rather than runs both Operator and Block.

Talked with @Superjom , he said we can do iterative polish to the design doc. If we merge this design doc, we need to address these two issues ASAP.

…esign.md

JiayiFeng · 2017-09-12T19:03:53Z

doc/design/block.md

+- help to execute multiple operators, blocks should group operators and runs like a single operator.
+
+## How to use Block
+In `RNNOp`, `SwitchOp`, `WhileOp` and `IfElseOp`, a sub-block should be used to help to define a sub-block.


“a sub-block should be used to help to define a sub-block”

It's hard to understand...

…esign.md

wangkuiyi · 2017-09-17T02:26:39Z

doc/design/block.md

+W = pd.Variable(shape=[20, 20])
+U = pd.Varable(shape=[20, 20])
+
+rnn = create_rnn()


No input specified here.

add block

ea9eed0

Superjomn requested review from reyoung, jacquesqiao, qingqing01, JiayiFeng and QiJune August 27, 2017 21:53

QiJune reviewed Aug 28, 2017

View reviewed changes

wangkuiyi reviewed Aug 28, 2017

View reviewed changes

Superjomn added 3 commits August 30, 2017 09:59

add scope

7496f8b

add some interfaces

e69aa44

Merge branch 'block_design.md' of github.com:Superjom/Paddle into blo…

f1da88d

…ck_design.md

Superjomn commented Aug 30, 2017

View reviewed changes

Superjomn added 6 commits September 2, 2017 13:30

rename VarDescLib to VarDescScope

6448339

add SetScope interface

124dd0c

fix most spells

e55a3d8

add functions

d84bde3

update

8ab4951

update

2253219

wangkuiyi reviewed Sep 3, 2017

View reviewed changes

wangkuiyi mentioned this pull request Sep 3, 2017

Check variables were defined during block parsing #3832

Closed

Superjomn added 5 commits September 4, 2017 10:46

rewrite block design

da4f2a6

Merge branch 'block_design.md' of github.com:Superjom/Paddle into blo…

64ed5dd

…ck_design.md

fix grammer

247f4a9

fix grammer

6e62cb2

move ScopeInit to protected

c59e697

wangkuiyi previously approved these changes Sep 4, 2017

View reviewed changes

update

3f3de4a

Superjomn dismissed wangkuiyi’s stale review via 3f3de4a September 4, 2017 19:19

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into block_d…

6b6679a

…esign.md