Add soft-label support for cross-entropy operator. #4081

xinghai-sun · 2017-09-13T14:30:20Z

Resolve #4080
Resolve #3898

qingqing01

Need to update and fix conflicts.

qingqing01 · 2017-09-15T04:57:47Z

paddle/operators/cross_entropy_op.cc

-    auto *X = ctx.Input<Tensor>("X");
-    auto *label = ctx.Input<Tensor>("label");
+    auto *x = ctx.Input<Tensor>("X");
+    auto *label = ctx.Input<Tensor>("Label");


Please add not null check for Input(X) and Input(Label). Thanks!

qingqing01 · 2017-09-15T05:01:55Z

paddle/operators/cross_entropy_op.cc

@@ -17,59 +17,71 @@ limitations under the License. */
 namespace paddle {
 namespace operators {

-class OnehotCrossEntropyOp : public framework::OperatorWithKernel {
+class CrossEntropyOp : public framework::OperatorWithKernel {


@luotao1 changes CrossEntropyOp -> OnehotCrossEntropyOp. Please use the new name.

Since it supports not only one-hot cross-entropy but also soft-label cross-entropy, it would be better to use CrossEntropyOp instead of OnehotCrossEntropyOp.

qingqing01 · 2017-09-15T05:03:25Z

paddle/operators/cross_entropy_op.cc

+      // normal cross entropy
+      PADDLE_ENFORCE_EQ(x->dims()[0], label->dims()[0]);
+    }
+    ctx.Output<Tensor>("Y")->Resize({x->dims()[0]});


Output<framework::LoDTensor>

Now must use Output<framework::LoDTensor> for output in forward and backward InferShape.

@qingqing01 是否需要按照新的命名规范将输出 Y 改为 Out ？（之前益群在FC中使用Y作为输出时，也被要求改为 Out ，是否需要统一？）

After discussing with @Xreki , we both prefer "Loss" as the output name rather than "Out". I think Loss is more meaningful than "Out".

qingqing01 · 2017-09-15T05:03:49Z

paddle/operators/cross_entropy_op.cc

 public:
  using framework::OperatorWithKernel::OperatorWithKernel;

 protected:
  void InferShape(const framework::InferShapeContext &ctx) const override {
-    auto dX = ctx.Output<Tensor>(framework::GradVarName("X"));
-    auto X = ctx.Input<Tensor>("X");
+    auto dx = ctx.Output<Tensor>(framework::GradVarName("X"));


Output< framework::LoDTensor>

qingqing01 · 2017-09-15T05:07:23Z

paddle/operators/cross_entropy_op.cc

-    auto dX = ctx.Output<Tensor>(framework::GradVarName("X"));
-    auto X = ctx.Input<Tensor>("X");
+    auto dx = ctx.Output<Tensor>(framework::GradVarName("X"));
+    auto x = ctx.Input<Tensor>("X");


Also add not null check for Input(X). Thanks!

qingqing01 · 2017-09-15T05:09:16Z

paddle/operators/cross_entropy_op.cc

-                Y[i] = -log(X[i][j])
+The second input (Label tensor) supports two kinds of shapes:
+1) Rank(Label) = 1, Label[i] indicates the class index for sample i:
+                Y[i] = -log(X[i, Label[i]])


Add space before and after formula.

qingqing01 · 2017-09-15T05:09:24Z

paddle/operators/cross_entropy_op.cc


+2) Rank(Label) = 2, Label[i, j] indicates the soft label of class j
+   for sample i:
+                Y[i] = \sum_j{-Label[i, j] * log(X[i, j])}


Add space before and after formula.

qingqing01 · 2017-09-15T05:11:41Z

paddle/operators/cross_entropy_op.cu

@@ -21,17 +21,16 @@ namespace operators {
 using Tensor = framework::Tensor;

 template <typename T>
-__host__ __device__ T clipping_log(const T x) {
+__host__ __device__ T tolerable_value(const T x) {


include paddle/platform/hostdevice.h, then use HOSTDEVICE .

HOSTDEVICE T tolerable_value(const T x) {

I have a question here, if this function use __host__ __device__ in the declearation, why we need to implement it again in *.cc ?

如果换成 HOSTDEVICE，依据HOSTDEVICE的定义：

#ifdef __CUDACC__ #define HOSTDEVICE __host__ __device__ #define HOST __host__ #else #define HOSTDEVICE #define HOST #endif

确实cpu、gpu可以公用这个tolerable_value。

嗯~ 明白啦~ HOSTDEVICE 在 CPU 下为空。

qingqing01 · 2017-09-15T05:36:11Z

python/paddle/v2/framework/tests/test_cross_entropy_op.py

+        self.check_output()
+
+    def test_check_grad(self):
+        self.check_grad(['X'], 'Y', max_relative_error=0.05)


Could tune the max_relative_error smaller?

lcy-seso · 2017-09-16T09:55:55Z

当label变成soft时，已经不再会有离散化的操作，是否可以直接调用 Egien？

xinghai-sun

All done. Thanks.

xinghai-sun · 2017-09-16T10:07:32Z

paddle/operators/cross_entropy_op.cc

-    auto *X = ctx.Input<Tensor>("X");
-    auto *label = ctx.Input<Tensor>("label");
+    auto *x = ctx.Input<Tensor>("X");
+    auto *label = ctx.Input<Tensor>("Label");


xinghai-sun · 2017-09-16T10:07:45Z

paddle/operators/cross_entropy_op.cc

+      // normal cross entropy
+      PADDLE_ENFORCE_EQ(x->dims()[0], label->dims()[0]);
+    }
+    ctx.Output<Tensor>("Y")->Resize({x->dims()[0]});


xinghai-sun · 2017-09-16T10:07:55Z

paddle/operators/cross_entropy_op.cc

 public:
  using framework::OperatorWithKernel::OperatorWithKernel;

 protected:
  void InferShape(const framework::InferShapeContext &ctx) const override {
-    auto dX = ctx.Output<Tensor>(framework::GradVarName("X"));
-    auto X = ctx.Input<Tensor>("X");
+    auto dx = ctx.Output<Tensor>(framework::GradVarName("X"));


xinghai-sun · 2017-09-16T10:44:34Z

paddle/operators/cross_entropy_op.cc

-    auto dX = ctx.Output<Tensor>(framework::GradVarName("X"));
-    auto X = ctx.Input<Tensor>("X");
+    auto dx = ctx.Output<Tensor>(framework::GradVarName("X"));
+    auto x = ctx.Input<Tensor>("X");


xinghai-sun · 2017-09-16T10:44:54Z

paddle/operators/cross_entropy_op.cc

-                Y[i] = -log(X[i][j])
+The second input (Label tensor) supports two kinds of shapes:
+1) Rank(Label) = 1, Label[i] indicates the class index for sample i:
+                Y[i] = -log(X[i, Label[i]])


xinghai-sun · 2017-09-16T10:47:21Z

paddle/operators/cross_entropy_op.cu

@@ -21,17 +21,16 @@ namespace operators {
 using Tensor = framework::Tensor;

 template <typename T>
-__host__ __device__ T clipping_log(const T x) {
+__host__ __device__ T tolerable_value(const T x) {


xinghai-sun · 2017-09-16T10:52:59Z

python/paddle/v2/framework/tests/test_cross_entropy_op.py

+        self.check_output()
+
+    def test_check_grad(self):
+        self.check_grad(['X'], 'Y', max_relative_error=0.05)


xinghai-sun · 2017-09-16T10:56:38Z

@lcy-seso 我觉得也是可以的，如果愿意容忍同一个if的两个分支：一个走cuda代码，一个走eigen。

lcy-seso · 2017-09-16T10:59:43Z

caffe2 分为两个分支，可能更关键的是两者在计算上那个更高效。暂时未知。

lcy-seso · 2017-09-17T10:54:38Z

paddle/operators/cross_entropy_op.cu

+  // TOOD(qingqing) define CUDA_1D_KERNEL_LOOP macro in a common file.
+  // CUDA_1D_KERNEL_LOOP(i, N) {
+  for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < N;
+       i += blockDim.x * gridDim.x) {


我的小疑问请教一下 @qingqing01

kernel 函数里面的这个for循环，计算结果不会出错，但逻辑上感觉很怪。

grid中总会有一些block的thread是多余出来，不会严格对齐到输入数据，这里for循环的作用相当于跳过这些没有对齐的部分。

i += blockDim.x * gridDim.x 循环变量i只要增加一次，就会直接超过最大thread数，也就是这个kernel函数实际是并不需要循环多次，只计算输出向量的一个位置，逻辑上等价于判断i>= batch_size时，直接return，写成循环有什么考虑呢？

如果针对下面设置grid, threadd的方式，确实不需要for循环。但如果将下面的grid的设置为一个固定的数，就是总共发起固定数目的总线程数，for循环就是有用的，有可能一个线程计算多个输出。这样这个kernel已经处理了边界，就不需要修改了。

int block = 512; int grid = (n + block - 1) / block;

明白啦~ 确实，cross entropy 这个 kernel 比较简单也比较特殊。grid 数目也已经计算好。

qingqing01 · 2017-09-18T06:40:14Z

paddle/operators/cross_entropy_op.cc

+    auto *label = ctx.Input<Tensor>("Label");
+
+    PADDLE_ENFORCE_EQ(x->dims().size(), 2, "X's rank must be 2.");
+    PADDLE_ASSERT(label->dims().size() == 1 || label->dims().size() == 2);


As discussed in this morning, we should also use the label with rank 2 for the int label. Please help to modify it. And if so, there is no way to determine whether it is normal cross entropy or soft cross entropy by rank. Can we change to use attr?

qingqing01 · 2017-09-18T06:50:47Z

paddle/operators/cross_entropy_op.cu

+using Tensor = framework::Tensor;
+
+template <typename T>
+HOSTDEVICE T tolerable_value(const T x) {


As @lcy-seso said, both CPU and GPU kernel can use this common function, if we use HOSTDEVICE. Use this function to replace it in paddle/operators/cross_entropy_op.h file and delete this function in paddle/operators/cross_entropy_op.cc file.

qingqing01 · 2017-09-18T07:18:56Z

paddle/operators/cross_entropy_op.cu

+      sum += label[i * D + j] * log(X[i * D + j]);
+    }
+    Y[i] = -tolerable_value(sum);
+  }


Put tolerable_value aftern log:

for (int j = 0; j < D; j++) { sum += - label[i * D + j] * tolerable_value(log(X[i * D + j])); }

qingqing01 · 2017-09-18T07:29:04Z

paddle/operators/cross_entropy_op.cc

+CrossEntropy Operator.
+
+The second input (Label tensor) supports two kinds of shapes:
+1) Rank(Label) = 1, Label[i] indicates the class index for sample i:


If modify the int label rank, this doc comments also are needed to modify.

qingqing01 · 2017-09-18T07:33:09Z

paddle/operators/cross_entropy_op.cu

+  for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < N;
+       i += blockDim.x * gridDim.x) {
+    T sum = static_cast<T>(0);
+    for (int j = 0; j < D; j++) {


Please add todo optimization for this kernel.

qingqing01 · 2017-09-18T07:38:29Z

paddle/operators/cross_entropy_op.h

+        T sum = static_cast<T>(0);
+        for (int j = 0; j < class_num; ++j) {
+          sum += label_data[index] * std::log(x_data[index]);
+          y_data[i] = -tolerable_value(sum);


Use tolerable_value before std::log.

sum += - label_data[index] * tolerable_value(std::log(x_data[index]));

qingqing01 · 2017-09-19T12:41:42Z

Merge this PR, if there is any question, we will fix later.

Add soft-label support for cross-entropy operator.

6d60352

xinghai-sun requested a review from qingqing01 September 13, 2017 14:30

qingqing01 added the OpPorting label Sep 14, 2017

qingqing01 reviewed Sep 15, 2017

View reviewed changes

Merge branch 'develop' into soft_label_cross_entropy

d7717f2

xinghai-sun commented Sep 16, 2017

View reviewed changes

Update cross entropy operator by following reviewer's comments.

e870682

lcy-seso reviewed Sep 17, 2017

View reviewed changes

lcy-seso mentioned this pull request Sep 17, 2017

Softmax with cross entropy op. #4144

Merged

2 tasks

qingqing01 reviewed Sep 18, 2017

View reviewed changes

xinghai-sun added 2 commits September 19, 2017 15:23

Merge branch 'develop' into soft_label_cross_entropy

8e7fe8c

Use soft_label attribute for cross-entropy.

d8046da

qingqing01 previously approved these changes Sep 19, 2017

View reviewed changes

Fixed a error in mnist unitest.

19de8ae

xinghai-sun dismissed qingqing01’s stale review via 19de8ae September 19, 2017 11:20

qingqing01 approved these changes Sep 19, 2017

View reviewed changes

qingqing01 merged commit 5b42d2b into PaddlePaddle:develop Sep 19, 2017

Add soft-label support for cross-entropy operator. #4081

Add soft-label support for cross-entropy operator. #4081

Conversation

xinghai-sun commented Sep 13, 2017 • edited by lcy-seso Loading

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso commented Sep 16, 2017

xinghai-sun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinghai-sun commented Sep 16, 2017

lcy-seso commented Sep 16, 2017 • edited Loading

lcy-seso Sep 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Sep 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 commented Sep 19, 2017 • edited Loading

xinghai-sun commented Sep 13, 2017 •

edited by lcy-seso

Loading

lcy-seso Sep 18, 2017 •

edited

Loading

lcy-seso commented Sep 16, 2017 •

edited

Loading

lcy-seso Sep 17, 2017 •

edited

Loading

lcy-seso Sep 18, 2017 •

edited

Loading

qingqing01 commented Sep 19, 2017 •

edited

Loading