Deconv #269

wangyang59 · 2016-10-27T00:56:03Z

Implement CPU version of convolution transpose (deconv) layer using expand method.

qingqing01 · 2016-10-27T02:05:14Z

paddle/gserver/layers/ExpandConvTransLayer.cpp

+      expandInData += subK * subN;
+    }
+  }
+}


Actually, the forward of de-conv is the backward of convolution layer. And the backward is the forward of convolution layer. So it's better to reuse the implementation in convolution layer. This can refer the implementation method in Caffe.

The name DeConvLayer may be better.

Hi @qingqing01 Thanks a lot for your comments~

I did reuse most of the code from convolution layer and swap the forward and backward. However, there is still some subtle differences that keep me from directly using the original code in convolution layer. For example, the multiple inputs are on the convoluted side rather than the image side, and the biases are added on different sides too (you can compare ExpandConvLayer.cpp and ExpandConvTransLayer.cpp for details). So if we want to use the same code for both conv and deconv, we might need to refactor conv-layer too.

There was a discussion that convolution transpose might be more technically correct here. So tensorflow actually changed its name from "tf.nn.deconv2d" to "tf.nn.conv2d_transpose". You can also see the references I attached below.
Transpose convolution layer for tensorflow (was deconvolution) tensorflow/tensorflow#256 (comment)
http://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers

I see. It's better to refactor code. It might be able to extract the same code into forwardWeight/backwardWeight and forwardBias/backwardBias (or other suitable name), then call these functions in ExpandConvLayer and ExpandConvTransLayer.

wangyang59 · 2016-11-02T00:12:22Z

Hi @qingqing01
I have refactored the code following your suggestions and now ExpandConvLayer and ExpandConvTransLayer share a common superclass of ConvBaseLayerCpu which contains most of the shared codes.
I have also fixed a bug in the original ExpandConvLayer.bpropSharedBias
Could you please help take a look at it again? Thanks!

luotao1 · 2016-11-02T02:49:40Z

paddle/gserver/layers/ConvBaseLayer.h

@@ -27,6 +27,9 @@ class ConvBaseLayer : public Layer {
 protected:
  typedef std::vector<int> IntV;

+  /// True if it's convolution layer, false if it's deconv layer
+  bool isConv_;


变量名改成isDeConv或者别的，因为都是ConvLayer，不适合用isConv

luotao1 · 2016-11-02T03:03:56Z

paddle/gserver/layers/ConvBaseLayerCpu.h

+  /// The spatial dimensions of height of output feature map.
+  IntV outputH_;
+  /// The spatial dimensions of width of output feature map.
+  IntV outputW_;


imgSizeH_, imgSizeW_, outputH_, outputW_这四个变量在ConvBaseLayer里面有，可以直接继承，不用再写一遍了

The variables in ConvBaseLayer are imgSize_ outputX_ which does not distinguish width and height

Sorry, I didn't realize that there was a change by @qingqing01 made to ConvBaseLayer and ExpandConvLayer

luotao1 · 2016-11-02T03:05:50Z

paddle/gserver/layers/ConvBaseLayerCpu.h

+/**
+ * @brief A subclass of ConvBaseLayer that is a superclass of both
+ * ExpandConvLayer and ExpandConvTransLayer
+ */


既然是ExpandConvLayer and ExpandConvTransLayer的superclass，名字不要叫ConvBaseLayerCpu。可以叫ExpandConvBaseLayer，或者别的，名字要有所反应。

I have renamed it to ExpandConvBaseLayer

luotao1 · 2016-11-02T03:06:33Z

paddle/gserver/layers/ConvBaseLayerCpu.h

+
+  /*The expandInput_ and transOutValue_ are used for CPU expand conv calc*/
+  /// Expand one sample at a time. shape:
+  /// (numChannels * filterPixels_, outputSizeH * outputSizeW)


46-48行的注释，要不全用///，要不就用/**/

luotao1 · 2016-11-02T03:10:35Z

paddle/gserver/layers/ExpandConvLayer.cpp

@@ -166,7 +70,7 @@ void ExpandConvLayer::forward(PassType passType) {
    image = prevLayer->getOutputValue();
    for (size_t off = 0; off < image->getHeight(); off++) {
      REGISTER_TIMER_INFO("expandFwdOnce", getName().c_str());
-      expandFwdOnce(image, i, off);
+      expandFwdOnce(image, getOutputValue(), i, off);


在for循环外：MatrixPtr outV = getOutputValue(); for循环里面直接调用outV，可以少访问image->getHeight()次指针。

luotao1 · 2016-11-02T03:11:43Z

paddle/gserver/layers/ExpandConvLayer.cpp

@@ -218,109 +98,16 @@ void ExpandConvLayer::backward(const UpdateCallback &callback) {

  for (size_t i = 0; i != inputLayers_.size(); ++i) {
    /* First, calculate the input layers error */
-    bpropActs(outGrad, i);
+    if (NULL != getPrev(i)->getOutputGrad()) {


if语句改成：if (getPrev(i)->getOutputGrad())，多处需修改

luotao1 · 2016-11-02T03:16:41Z

paddle/gserver/layers/ExpandConvTransLayer.cpp

+  resetOutput(batchSize, getSize());
+
+  MatrixPtr output = nullptr;
+  for (size_t i = 0; i != inputLayers_.size(); ++i) {


for循环里面的终止条件建议写成 i < inputLayers_.size()，多处需修改

luotao1 · 2016-11-02T03:21:26Z

python/paddle/trainer/config_parser.py

@@ -1067,6 +1067,37 @@ def parse_conv(conv, input_layer_name, conv_conf):
            1 + int(math.ceil((2 * conv.padding + conv_conf.img_size \
            - conv.filter_size) / float(conv.stride)))

+
+def parse_convt(conv, input_layer_name, conv_conf, num_filters):


改成parse_conv_trans比较容易懂，实现的时候，1072-1098行有些可以复用parse_conv代码

luotao1 · 2016-11-02T03:24:22Z

python/paddle/trainer_config_helpers/layers.py

+@wrap_bias_attr_default()
+@wrap_act_default(act=ReluActivation())
+@layer_support(DROPOUT)
+def img_convTrans_layer(input, filter_size, num_filters,


img_convTrans_layer，能在img_conv_layer中加一个trans的变量来实现么

luotao1

merge conflict

luotao1 · 2016-11-02T03:35:49Z

paddle/gserver/tests/test_LayerGrad.cpp

@@ -262,6 +262,8 @@ void testConvLayer(const string& type, bool trans, bool useGpu) {
                              config.layerConfig.num_filters());

  testLayerGrad(config, "conv", 100, trans, useGpu);
+  // Use small batch_size and useWeight=true to test biasGrad
+  testLayerGrad(config, "conv", 2, trans, useGpu, true, 0.02);


batchsize=2也太小了，10或者100

The reason I set the batchsize to 2 is that the bug in calculating the bpropSharedBias cannot be caught with large batchsize due to averaging out errors.

请问有什么bug？可以贴出来么

void ExpandConvBaseLayer::bpropSharedBias(MatrixPtr biases, MatrixPtr v) {
size_t mapW = getOutputSize() / numFilters_;
size_t mapH = v->getElementCnt() / mapW;
MatrixPtr vTmp = Matrix::create(v->getData(), mapH, mapW, false, useGpu_);

Matrix::resizeOrCreate(transOutValue_, mapW, mapH, false, useGpu_);

vTmp->transpose(transOutValue_, false); // false means no memory allocation
transOutValue_->reshape(transOutValue_->getElementCnt() / numFilters_,
numFilters_);
biases->collectBias(*transOutValue_, 1.0f);
}

The last two lines was
vTmp->reshape(transOutValue_->getElementCnt() / numFilters_,
numFilters_);
biases->collectBias(*vTmp, 1.0f);

你的意思是改过后，就不会出bug？那可以修正下吧

wangyang59 · 2016-11-02T17:16:16Z

paddle/gserver/layers/ConvBaseLayer.h

+         (outputSize - 1) * stride + filterSize - 2 * padding - stride + 1;
+    } else {
+     imageSize = (outputSize - 1) * stride + filterSize - 2 * padding;
+    }


Here I feel that writing out two equations explicitly here is better

可以直接调用mathUtils.h中的outputSize函数（就多了最后一个caffeMode_的变量），不用在这里重新写一份。另外，把imageSize函数也放到mathUtils.h里吧

wangyang59 · 2016-11-02T23:14:36Z

Hi @luotao1 I have modified my codes per your suggestions.

@qingqing01 , when I rebase my code, I realized that you have also modified ConvBaseLayer and ExpandConvBaseLayer at the same time (pull#218). I have merged my changes with yours, so could you please also have a look at it to make sure that I didn't break your code? Thanks!

luotao1 · 2016-11-03T02:44:33Z

Note that 'All checks have failed', you can click 'Details' to see the log.

luotao1 · 2016-11-03T02:47:54Z

paddle/gserver/layers/ConvBaseLayer.h

@@ -74,6 +77,8 @@ class ConvBaseLayer : public Layer {
  /// of output size.
  bool caffeMode_;

+
+


多余的两行空行（80-81行）可以去掉

luotao1 · 2016-11-03T02:49:35Z

paddle/gserver/layers/ExpandConvBaseLayer.cpp

+   * convTrans, and in other functions too.
+   * */
+  int channel;
+  int nf;


nf变量改成numFilters，比较直接

luotao1 · 2016-11-03T02:51:20Z

paddle/gserver/layers/ExpandConvBaseLayer.cpp

+  /* Initialize the projection */
+  for (auto &inputConfig : config_.inputs()) {
+    const ConvConfig &conf = inputConfig.conv_conf();
+    nf = (!isDeconv_) ? numFilters_ : conf.channels();


nf = isDeconv_ ? conf.channels() : numFilters_; 41行也同样

luotao1 · 2016-11-03T03:10:27Z

python/paddle/trainer_config_helpers/layers.py

@@ -36,7 +36,7 @@
           "pooling_layer", "lstmemory", "last_seq", "first_seq",
           "cos_sim", "hsigmoid", "conv_projection",
           "regression_cost", 'classification_cost', "LayerOutput",
-           'img_conv_layer', 'img_pool_layer', 'batch_norm_layer',
+           'img_conv_layer', 'img_convTrans_layer', 'img_pool_layer', 'batch_norm_layer',


img_convTrans_layer去掉，不然导致单测不过

luotao1 · 2016-11-03T03:14:08Z

python/paddle/trainer/config_parser.py

@@ -1105,6 +1105,37 @@ def parse_conv(conv, input_layer_name, conv_conf):
            1 + int(math.ceil((2 * conv.padding + conv_conf.img_size \
            - conv.filter_size) / float(conv.stride)))

+
+def parse_conv_trans(conv, input_layer_name, conv_conf, num_filters):


parse_conv_trans和parse_conv的共用代码很多，能否在parse_conv里面加一个trans变量。同样ConvTransLayerBase和ConvLayerBase的共用代码也很多，能否在ConvLayerBase里面加一个trans变量。

done for the parse_conv_trans
However for the ConvTransLayerBase, I am not sure I can do the same thing since it has a decorator and the overlapping of the codes is not that big.

…rCpu

…s with pull#218 commit 45c81a4

…arser.py

…nv layer

coveralls · 2016-11-09T20:47:02Z

Coverage increased (+0.03%) to 62.474% when pulling 1c58e27 on wangyang59:deconv into 05204af on baidu:develop.

coveralls · 2016-11-09T23:00:47Z

Coverage increased (+0.03%) to 62.478% when pulling 1c58e27 on wangyang59:deconv into 05204af on baidu:develop.

wangyang59 · 2016-11-09T23:03:57Z

@luotao1 @qingqing01 The build on mac is not stable. Build #724 failed and build #725 passed although they have identical code base.

fix conv typo in develop

* debug nan * debug nan * skip weight_decay

* add docs for xlnet module introduction * add docs for xlnet module introduction

Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com>

* fix * fix * update * update

qingqing01 requested changes Oct 27, 2016

View reviewed changes

reyoung changed the base branch from master to develop October 27, 2016 02:10

qingqing01 modified the milestone: Image Classification.1 Oct 27, 2016

wangyang59 force-pushed the deconv branch from c6da895 to c6542a8 Compare November 1, 2016 23:53

gangliao assigned qingqing01 Nov 2, 2016

qingqing01 assigned luotao1 Nov 2, 2016

luotao1 reviewed Nov 2, 2016

View reviewed changes

luotao1 requested changes Nov 2, 2016

View reviewed changes

luotao1 reviewed Nov 2, 2016

View reviewed changes

wangyang59 commented Nov 2, 2016

View reviewed changes

wangyang59 force-pushed the deconv branch from 47e03e7 to fab16c5 Compare November 2, 2016 23:11

luotao1 reviewed Nov 3, 2016

View reviewed changes

wangyang59 added 14 commits November 9, 2016 11:43

Refactor ExpandConvTransLayer to share codes with ExpandConvLayer

aa2cd2c

refactored ExpandConvLayer and ExpandConvTransLayer with ConvBaseLaye…

2575b74

…rCpu

fixed a bug in refactoring ExpandConv/TransLayer

e68b50a

add another small test in test_LayerGrad for convTransLayer

5fff96f

Revised deconv implementations according to luotao1

5e4cc24

rebase deconv implementation with develop branch and resolve conflict…

3d72e94

…s with pull#218 commit 45c81a4

deconv layer implementation modification following luotao1 comments

fb20187

fix a small bug in ConvTransLayerBase in config_parser.py

d116b17

deconv implementation mionr changes in ConvBaseLayer.cpp and config_p…

7a322df

…arser.py

minor changes on deconv per luotao1 comments

03f4b1d

Refactored imageSize in ConvBaseLayer to MathUtil

53e1629

minor change to convTransLayer test in test_LayerGrad

4491209

minor changes on deconv implementation and add protostr test for deco…

af7a50c

…nv layer

fixed a bug in parse_conv in config_parser.py

1c58e27

wangyang59 force-pushed the deconv branch from a9c7686 to 1c58e27 Compare November 9, 2016 19:45

wangyang59 closed this Nov 9, 2016

wangyang59 reopened this Nov 9, 2016

luotao1 approved these changes Nov 10, 2016

View reviewed changes

qingqing01 approved these changes Nov 10, 2016

View reviewed changes

qingqing01 merged commit cfc965d into PaddlePaddle:develop Nov 10, 2016

qingqing01 mentioned this pull request Nov 11, 2016

No atrous convolution in conv layer ? #146

Closed

zhhsplendid pushed a commit to zhhsplendid/Paddle that referenced this pull request Sep 25, 2019

Merge pull request PaddlePaddle#269 from tink2123/cherry-pick_conv

c30a5ee

fix conv typo in develop

gglin001 pushed a commit to graphcore/Paddle-fork that referenced this pull request Dec 8, 2021

Debug nan (PaddlePaddle#269)

b43ebbf

* debug nan * debug nan * skip weight_decay

wangxicoding pushed a commit to wangxicoding/Paddle that referenced this pull request Dec 9, 2021

add docs for xlnet module introduction (PaddlePaddle#269)

d6665b7

* add docs for xlnet module introduction * add docs for xlnet module introduction

qingshui pushed a commit to qingshui/Paddle that referenced this pull request Apr 26, 2023

format code (PaddlePaddle#269)

a95fc66

Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com>

danleifeng pushed a commit to danleifeng/Paddle that referenced this pull request Sep 13, 2023

format code (PaddlePaddle#269)

d7c49b3

lizexu123 pushed a commit to lizexu123/Paddle that referenced this pull request Feb 23, 2024

fix something wrong about rlnas (PaddlePaddle#269)

fd248eb

* fix * fix * update * update

		@@ -74,6 +77,8 @@ class ConvBaseLayer : public Layer {
		/// of output size.
		bool caffeMode_;

Deconv #269

Deconv #269

Conversation

wangyang59 commented Oct 27, 2016

Choose a reason for hiding this comment

wangyang59 Oct 27, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyang59 commented Nov 2, 2016 • edited Loading

luotao1 Nov 2, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 Nov 2, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyang59 commented Nov 2, 2016

luotao1 commented Nov 3, 2016

luotao1 Nov 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 Nov 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Nov 9, 2016

coveralls commented Nov 9, 2016

wangyang59 commented Nov 9, 2016

wangyang59 Oct 27, 2016 •

edited

Loading

wangyang59 commented Nov 2, 2016 •

edited

Loading

luotao1 Nov 2, 2016 •

edited

Loading

luotao1 Nov 2, 2016 •

edited

Loading

luotao1 Nov 3, 2016 •

edited

Loading

luotao1 Nov 3, 2016 •

edited

Loading