Add ClipLayer #3112

guoshengCS · 2017-07-30T15:08:40Z

Add ClipLayer for clipping the input value by the threshold.

guoshengCS · 2017-07-30T15:13:33Z

不太清楚commit的时候为什么python/paddle/trainer_config_helpers/layers.py里__all__ = 的格式被调整了

qingqing01 · 2017-08-01T13:20:13Z

paddle/gserver/layers/ClipLayer.cpp

+  Matrix::resizeOrCreate(
+      tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_);
+  tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_);
+  inG->addDotMul(*outG, *tmpMtx, 1, 1);


if(inG) { MatrixPtr outV = getOutputValue(); MatrixPtr outG = getOutputGrad(); MatrixPtr tmpMtx; Matrix::resizeOrCreate( tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_); tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_); inG->addDotMul(*outG, *tmpMtx, 1, 1); }

~~反向的计算不对吧？梯度直接往前层传递，不需要对梯度截断。~~
抱歉，我犯蠢了，梯度计算是正确的。

前向clip之后，反向需要截断： https://github.com/caffe2/caffe2/blob/master/caffe2/operators/clip_op.cc#L29

好的，明白啦。我犯蠢了，这里计算的是如果将clip当做一种运算，clip 这运算的导数：超过上下阈值部分输出是常数，对$x$求导梯度为0；否则对$x$求导，梯度为1。

$$y=\text{clip}(x)$$

$$\text{x} = \left{ \begin{align} &a &x< a \\ &x &a \le x \le b \\ &b &x <b \end{align}\right.$$

$$\frac{\partial{L}}{\partial{x}} = \frac{\partial{L}}{\partial{y}}\frac{\partial{y}}{\partial{x}}$$ $$\frac{\partial{y}}{\partial{x}}=\left{ \begin{align} &1 &a \le x \le b \\ &0 &\text{else} \end{align}\right.$$

对的，clipDerivative计算的是clip的(sub)gradient

qingqing01 · 2017-08-01T13:24:37Z

python/paddle/trainer/config_parser.py

@@ -2169,6 +2169,23 @@ def __init__(self, name, inputs, context_length, **xargs):
        self.create_input_parameter(0, psize, dims)


+@config_layer('clip')
+class ClipLayer(LayerBase):
+    def __init__(self, name, inputs, clip_threshold_low, clip_threshold_high):


def __init__(self, name, inputs, clip_threshold_low, clip_threshold_high, **xargs):

Done.已添加

qingqing01 · 2017-08-01T13:26:36Z

proto/ModelConfig.proto

@@ -289,6 +289,11 @@ message DetectionOutputConfig {
  optional uint32 width = 9 [default = 1];
 }

+message ClipConfig {
+  required float clip_threshold_low = 1;
+  required float clip_threshold_high = 2;


maybe double is better.

clip_threshold_low -->min

clip_threshold_high-->max

Done，已将type和name进行修改

qingqing01 · 2017-08-01T13:27:47Z

python/paddle/trainer_config_helpers/layers.py

+        clip_threshold_low=clip_threshold_low,
+        clip_threshold_high=clip_threshold_high)
+    return LayerOutput(
+        name, LayerType.CLIP_LAYER, parents=[input], size=input.size)


unit test for this interface

add this layer in the doc: https://github.com/PaddlePaddle/Paddle/blob/develop/doc/api/v2/config/layer.rst

Done.已添加

lcy-seso · 2017-08-01T13:59:28Z

paddle/gserver/layers/ClipLayer.cpp

+ * \f]
+ */
+
+class ClipLayer : public Layer {


ClipLayer 重命名成 ValueClipLayer

ClipLayer已经是个Layer了，forward一定是value的操作，所以其实觉得ClipLayer名字还好

好的，同意。

Done，使用ClipLayer命名

lcy-seso · 2017-08-01T14:01:56Z

paddle/gserver/layers/ClipLayer.cpp

+class ClipLayer : public Layer {
+protected:
+  real clipThresholdLow_;
+  real clipThresholdHigh_;


clipThresholdLow_ --> min_

clipThresholdHigh_ --> max_

还有一个问题，一般这种截断都是对称截断，因此不需要upper、lower 两个阈值，请问一定需要两个阈值吗？

觉得两个阈值功能多些，caffe2的clip_op也是min,max两个阈值： https://github.com/caffe2/caffe2/blob/master/caffe2/operators/clip_op.h#L55

好的，明白啦。同意。

Done，已修改为min_和max_

lcy-seso · 2017-08-01T14:10:21Z

paddle/gserver/layers/ClipLayer.cpp

+  Matrix::resizeOrCreate(
+      tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_);
+  tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_);
+  inG->addDotMul(*outG, *tmpMtx, 1, 1);


~~反向的计算不对吧？梯度直接往前层传递，不需要对梯度截断。~~
抱歉，我犯蠢了，梯度计算是正确的。

lcy-seso · 2017-08-01T14:11:31Z

proto/ModelConfig.proto

@@ -289,6 +289,11 @@ message DetectionOutputConfig {
  optional uint32 width = 9 [default = 1];
 }

+message ClipConfig {
+  required float clip_threshold_low = 1;
+  required float clip_threshold_high = 2;


clip_threshold_low -->min

clip_threshold_high-->max

… add-ClipLayer

lcy-seso

LGTM.

add ClipLayer

23a8d01

guoshengCS requested review from lcy-seso, qingqing01 and livc July 30, 2017 15:08

revise the format of __all__ in layers.py

f35e084

qingqing01 reviewed Aug 1, 2017

View reviewed changes

lcy-seso requested changes Aug 1, 2017

View reviewed changes

guoshengCS added 3 commits August 2, 2017 15:50

Refine ClipLayer and add unit test for it

d19355a

Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…

82911ce

… add-ClipLayer

fix bug on ClipLayer test in test_LayerGrad

d1a6917

lcy-seso approved these changes Aug 2, 2017

View reviewed changes

Add test_clip_layer.protostr for ClipLayer python unit test

5d64499

qingqing01 approved these changes Aug 2, 2017

View reviewed changes

Merge branch 'develop' into add-ClipLayer

b7b956f

guoshengCS merged commit 0c181c6 into PaddlePaddle:develop Aug 3, 2017

heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021

fix doc, set min_subgraph_size of s2anet (PaddlePaddle#3112)

8c7126b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ClipLayer #3112

Add ClipLayer #3112

guoshengCS commented Jul 30, 2017

guoshengCS commented Jul 30, 2017

qingqing01 Aug 1, 2017

lcy-seso Aug 1, 2017 •

edited

Loading

qingqing01 Aug 1, 2017 •

edited

Loading

lcy-seso Aug 2, 2017 •

edited

Loading

guoshengCS Aug 2, 2017

qingqing01 Aug 1, 2017

guoshengCS Aug 2, 2017

qingqing01 Aug 1, 2017

lcy-seso Aug 1, 2017 •

edited

Loading

guoshengCS Aug 2, 2017

qingqing01 Aug 1, 2017

guoshengCS Aug 2, 2017

lcy-seso Aug 1, 2017

qingqing01 Aug 1, 2017

lcy-seso Aug 2, 2017

guoshengCS Aug 2, 2017

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso Aug 1, 2017

qingqing01 Aug 1, 2017

lcy-seso Aug 2, 2017 •

edited

Loading

guoshengCS Aug 2, 2017

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso left a comment

Add ClipLayer #3112

Add ClipLayer #3112

Conversation

guoshengCS commented Jul 30, 2017

guoshengCS commented Jul 30, 2017

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

qingqing01 Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

lcy-seso Aug 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Aug 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

lcy-seso left a comment

Choose a reason for hiding this comment

lcy-seso Aug 1, 2017 •

edited

Loading

qingqing01 Aug 1, 2017 •

edited

Loading

lcy-seso Aug 2, 2017 •

edited

Loading

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso Aug 2, 2017 •

edited

Loading

lcy-seso Aug 1, 2017 •

edited

Loading

lcy-seso Aug 1, 2017 •

edited

Loading