Python v2 API to new operator framework #3129
Comments
|
How to serialize/deserialize network topology -- @reyoung It seems that we can write a function |
|
Make sure that the traced-and-built network in this design supports graph visualization -- @zchen0211 I am sure it supports. -- @wangkuiyi |
|
Change RNN step-net input placeholder from NULL to |
|
The parameter |
|
To support memory in RNN, we might need Placeholder as a special type. -- @zchen0211 @reyoung |
|
Need to consider the initialization of parameters. -- @wangkuiyi |
|
Need to consider parameter sharing and enable/disable parameter update, so could we implement GAN. -- @wangkuiyi |
|
I think make struct OperatorBase {
vector<string> inputs_;
vector<string> outputs_;
};
struct Expression {
// All operators to calculate that expression. So the operator could be a NetOp.
OperatorBase* operator;
size_t output_index;
}Each function in Python should return an def data_layer(data_type, reader):
op = paddle.cpp.new_op("data", reader=reader, data_type=data_type)
return Expression(op, 0)
def fc_layer(input, size, ...):
w = paddle.cpp.new_var("w", ...)
b = paddle.cpp.new_var("b", ...)
out = paddle.cpp.new_var("out", ...)
net = paddle.cpp.new_net()
net.add_op(input.op) # prepend previous operators here.
op = paddle.cpp.new_op("fc", input=[
input.op.outputs[input.op.output_index], # could extract a helper function here.
w,
b,
], output=[out])
net.add_op(op)
return Expression(net, 0)When getting the whole network of an operator, it could just data = data_layer(...)
hidden = fc_layer(data, ...) # hidden.operator contains data->fc
cost = fc_layer(hidden, ...) # cost.operator contains data->fc->cost
cost.operator # cost's network.Also, we could add some helper function into class Expression {
public:
Variable* calculate_var(const Scope& scope) {
operator->Run(scope);
return to_var(scope);
}
Variable* to_var(const Scope& scope) {
return scope->find_var(operator->outputs[output_index]);
}
OperatorBase* operator;
size_t output_index;
}; |
|
@wangkuiyi Says to introduce an @reyoung thinks that we could use |
|
@reyoung This design in #3129 (comment) introduces class Also, it requires those researchers who program layers to provide connection information; instead of using For above two reasons, I don't think it is better than #3129 (comment). |
|
may be The user views an operator's inputs and outputs as a concept of arguments, so we can wrap the details and give a corresponding concept. |
|
It would be great if we can have a simple enough way to hide the details from users. Let us write the code so to present a comparison. @Superjom |
|
Both the The python-wrapper is much cheap to modify, so I tried to move the codes above from c++ to python, and try to prove that
If it is not a good idea to move those codes from C++ to Python, ignore following codes ... I implement # original Variable
import paddle.cpp.Variable as CPPVar
class Variable:
'''
A wrapper for Variable defined in CPP, and leave net trace
or expression in python.
'''
def __init__(self, cpp_var):
'''
@cpp_var: CPPVar
'''
self._cpp_var = cpp_var
self._read_ops = set()
self._writs_ops = set()
# a net op which stores the subnets.
self.subnet = None
def cpp_var(self):
return self._cpp_var
def add_read_op(self, cpp_op):
self._read_ops.add(cpp_op)
def add_write_op(self, cpp_op):
self._write_ops.add(cpp_op)
def __repr__(self):
'''
acts exactly like a Variable
'''
return repr(self._cpp_var)
class Scope:
def __init__(self):
self._cpp_scope = paddle.cpp.new_scope()
# store Variable map
self._var_map = {}
def new_var(self):
cpp_var = self._cpp_scope.new_var()
v = Variable(cpp_var)
self._var_map[v.name] = v
def get_var(self, label):
return self._var_map[label]
def new_op(op_type, reads, writes, **attrs):
'''
detect variable dependencies.
'''
cpp_op = paddle.cpp.operator(op_type, inputs=[v.cpp_var() for v in reads],
outputs=[v.cpp_var() for v in writes], **attrs)
for v in reads:
v.add_read_op(cpp_op)
for v in writes:
v.add_write_op(cpp_op)
return cpp_op
default_scope = Scope()
def fc(input):
'''
@input: Variable
@output: Variable
'''
output = default_scope.new_var()
W = default_scope.get_var(label)
b = default_scope.get_var(label)
new_op("FC", reads=[input, W, b], output_size, writes=[output])
return output
def backtrace(var):
'''
the implementation by @wangyi in c++ before, we can move this to python.
batcktrace variable and create a subnet(NetOp).
'''
# much details here
pass
def paddle_run(var):
'''
@var: Variable
run a subnet whose end point is var, if this network contains cost var,
then backward will be called, otherwise just forward.
'''
if not var.subnet:
var.subnet = backtrace(var)
var.subnet.Run()
def serialize(var):
'''
@var: Variable
'''
if not var.subnet:
var.subnet = backtrace(var)
return var.subnet.Serialize() |
|
@Superjom If I understand this correctly, it would be much much less code to add Let us focus on more important topics like parameter sharing -- those not yet fixed in this design. It doesn't look like we can really have an easier solution to tracing the topology than the first proposal. However, the first proposal is not verified to work with model sharing yet. |
wangkuiyi
changed the title
Design Doc: RNNOp
A Plain Network
Layers, Variables, and Default Scope
where
paddle.cpp.variableis a Python binding of C++ methodScope::NewVar().paddle.cpp.operatorcreates an operator and mark it as a reader of some variable and a writer of some others. We will cover this later in more details.paddle::operator::Netpaddle.trainreceives a variable created bypaddle.layer.mseand need to trace all related operators and sort them by the topological order.Please be aware that all operators are derived from class
OperatorBase, which refers to Variables by their names:and Variables doesn't have names if they are not in a Scope.
Also, each Varaible maintains:
Please be aware the trace from an operator to its input variables depends on the default scope. The tracing is done in C++ space, so
paddle.cpp.default_scopeis a binding to C++ code.We can call
Net::TraceAndBuild(output_variable, DefaultScope()).Run(DefaultScope());to extract the network using the default scope and run it.
Scope Hierarchy
An RNN operator may have kinds of variables:
Just like what a programing language compiler/interpreter would do, for each step, there is a step-local scope, but there is only one copy of compiled code (binary code) or step-net in our case.
Above three tiers can be simplified to two-tier by moving memory variables to the outer scope, but this is not necessary.
A Recurrent Network
Here we use
NULLas the placeholder of the input of the step net.Please notice that we don't have to consume the output of an RNNOp. For example, we can use the memory as the RNNOp's output:
Step-Net
Above example shows that the
step_netparameter ofpaddle.layer.rnnaccepts a variable returned bypaddle.layer.fc. We need to trace the step-net from this variable. This can be done by calling the aforementinedpaddle::operator::Net::TraceAndBuildThe text was updated successfully, but these errors were encountered: