[WIP] Backward #2949

dzhwinter · 2017-07-18T13:06:05Z

naive implement of backward op. Need @Canpio add reorder of input index.

… develop

Superjomn · 2017-07-18T13:07:44Z

paddle/framework/fully_connected_op.cc

+   limitations under the License. */
+
+#include "paddle/framework/fully_connected_op.h"
+#include <iostream>


#include <iostream> #include "paddle/framework/fully_connected_op.h"

…ackward

We put forward Op's inputs, outputs and output gradients into Grad Op's inputs, and put forward Op's input gradients into Grad Op's output. So Grad Op's `in_out_idx`, `input_format` and 'output format' need to be rebuilt during Op creating.

reyoung · 2017-07-19T05:37:04Z

paddle/framework/op_registry.h

+ */
+#define REGISTER_GRADIENT_OP(__op_type, __op_class)            \
+  STATIC_ASSERT_GLOBAL_NAMESPACE(                              \
+      __reg_op__##__op_type,                                   \


这个名字可以再特殊一些，例如

__reg_gradient_op__##__op_type,

否则比较容易和其他的REGISTER冲突。

reyoung · 2017-07-19T05:37:13Z

paddle/framework/op_registry.h

+      __reg_op__##__op_type,                                   \
+      "REGISTER_GRADIENT_OP must be in global namespace");     \
+  static ::paddle::framework::GradOpRegisterHelper<__op_class> \
+      __op_register_##__op_type##__(#__op_type);               \


同理，这个名字也可以特殊一些。。

reyoung · 2017-07-19T05:38:41Z

paddle/framework/op_registry.h

@@ -274,6 +281,18 @@ class OpRegistry {
    return CreateOp(op_desc.type(), inputs, outputs, attrs);
  }

+  static OperatorPtr CreateGradOp(OperatorPtr op) {
+    OperatorPtr grad_op(grad_creators().at(op->type_)());


If not found that Gradient Op，just return nullptr is cool

reyoung · 2017-07-19T05:39:17Z

paddle/framework/net.h

@@ -101,5 +101,7 @@ class PlainNet : public Net {
  }
 };

+std::shared_ptr<PlainNet> AddBackwardOp(std::shared_ptr<PlainNet> ForwardOps);


I think this method might be useless, we can directly invoke OpRegistry::CreateGradOp(...) even that Op is a net.

That's true. I remembered that AddBackwardOp is an interface for PlainNet, which generate the backward ops, then it can be used in Python. Remove it straightly is fine to me.

JiayiFeng · 2017-07-19T06:24:03Z

@reyoung reminded us that not all inputs/outputs are required by Op's gradient calculating. So we are going to refactor related functions to support flexible Grad Op inputs/outputs generating.

…ackward

Superjomn · 2017-07-20T04:49:54Z

paddle/framework/op_registry.h

@@ -274,6 +285,24 @@ class OpRegistry {
    return CreateOp(op_desc.type(), inputs, outputs, attrs);
  }

+  static OperatorPtr CreateGradOp(OperatorPtr op) {


CreateGradientOp ?

we named every Grad operator in XXXGrad, so we create interface named in CreateGradOp

Superjomn · 2017-07-20T04:52:34Z

paddle/framework/op_registry.h

-  static std::unordered_map<std::string, OpCreator>& creators() {
-    static std::unordered_map<std::string, OpCreator> creators_;
-    return creators_;
+  static void AssembleGradInOut(OperatorPtr op, OperatorPtr grad_op) {


AssembleInOutGradient?

Superjomn · 2017-07-20T05:05:06Z

paddle/framework/operator.h

@@ -63,6 +63,11 @@ class OperatorBase {
  /// but it will be convert to a unique name in scope after OpCreator.
  static std::string TMP_VAR_NAME() { return "@TEMP@"; }


a static const member better? no need a getter

const static member can not ensure the calling order. See the effective C++ rule.

…ackward

…ward

… backward

Replace `OperatorPtr` with `std::shared_ptr<OperatorBase>`

JiayiFeng

LGTM

JiayiFeng and others added 6 commits July 2, 2017 17:10

remove simple_op_design.md

f35c8c4

renew simple_op_design.md

7dc53ea

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

015ccd4

… develop

"add backward Op"

816b4c8

fix confilct

9890b23

"add net op testing"

8b80cf8

Superjomn reviewed Jul 18, 2017

View reviewed changes

JiayiFeng and others added 3 commits July 18, 2017 22:29

test collaborating

7f1533f

move unused file

855cae6

Merge branch 'backward' of https://github.com/dzhwinter/Paddle into b…

e786746

…ackward

JiayiFeng changed the title ~~Backward~~ [WIP] Backward Jul 18, 2017

Refactor Rigistry::CreateGradOp()

bf4da3d

We put forward Op's inputs, outputs and output gradients into Grad Op's inputs, and put forward Op's input gradients into Grad Op's output. So Grad Op's `in_out_idx`, `input_format` and 'output format' need to be rebuilt during Op creating.

reyoung reviewed Jul 19, 2017

View reviewed changes

dzhwinter and others added 2 commits July 19, 2017 14:28

"ignore some gradient of specific op"

cb95587

rename a macro

94a6b1f

dzhwinter force-pushed the backward branch from ec55e3f to cb95587 Compare July 19, 2017 06:29

dzhwinter and others added 10 commits July 19, 2017 14:40

"fix comment "

8bc4892

Merge branch 'backward' of https://github.com/dzhwinter/Paddle into b…

3dc70ff

…ackward

Merge remote-tracking branch 'origin/develop' into backward2

73f4779

"make plainNet shared"

4876f35

Refactor the implementation of gradient Op creating

e192d0f

"use built-in operator"

14424f3

"test fc without gradient"

81a352a

Merge remote-tracking branch 'origin/develop' into backward2

6f05392

Fix some compile errors

8a5ee46

Fix some compile error

b635af7

JiayiFeng and others added 2 commits July 20, 2017 11:02

Merge branch 'backward' of https://github.com/dzhwinter/Paddle into b…

45452ac

…ackward

"remove unused fake fc op"

088e220

Superjomn reviewed Jul 20, 2017

View reviewed changes

JiayiFeng added 10 commits July 20, 2017 13:38

Merge branch 'backward_dev' into backward

99a5904

Merge branch 'backward' of https://github.com/dzhwinter/Paddle into b…

f41fcd4

…ackward

Fix compile errors

9418717

Add a simple test for grad_op_creator

4736b23

Renew CMakeList dependence

f85ccdd

Merge branch 'backward' of https://github.com/Canpio/Paddle into back…

0ab8f52

…ward

Renew CMakeList dependence

380227b

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

5f3bc2a

… backward

Fix compile error

f4e2555

Replace `OperatorPtr` with `std::shared_ptr<OperatorBase>`

fix compile errer

81df39f

JiayiFeng self-requested a review July 24, 2017 11:27

JiayiFeng approved these changes Jul 24, 2017

View reviewed changes

JiayiFeng merged commit 02cde24 into PaddlePaddle:develop Jul 24, 2017

reyoung mentioned this pull request Jul 25, 2017

Refine depends of op_registry #3038

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Backward #2949

[WIP] Backward #2949

dzhwinter commented Jul 18, 2017 •

edited

Loading

Superjomn Jul 18, 2017

reyoung Jul 19, 2017

JiayiFeng Jul 19, 2017

reyoung Jul 19, 2017

dzhwinter Jul 20, 2017

reyoung Jul 19, 2017

dzhwinter Jul 20, 2017

reyoung Jul 19, 2017 •

edited

Loading

dzhwinter Jul 20, 2017

JiayiFeng commented Jul 19, 2017

Superjomn Jul 20, 2017

dzhwinter Jul 20, 2017

Superjomn Jul 20, 2017

dzhwinter Jul 20, 2017

Superjomn Jul 20, 2017

dzhwinter Jul 20, 2017 •

edited

Loading

JiayiFeng left a comment

		@@ -63,6 +63,11 @@ class OperatorBase {
		/// but it will be convert to a unique name in scope after OpCreator.
		static std::string TMP_VAR_NAME() { return "@TEMP@"; }

[WIP] Backward #2949

[WIP] Backward #2949

Conversation

dzhwinter commented Jul 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyoung Jul 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiayiFeng commented Jul 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dzhwinter Jul 20, 2017 • edited Loading

Choose a reason for hiding this comment

JiayiFeng left a comment

Choose a reason for hiding this comment

dzhwinter commented Jul 18, 2017 •

edited

Loading

reyoung Jul 19, 2017 •

edited

Loading

dzhwinter Jul 20, 2017 •

edited

Loading