Fix dropout functor error #9808

clackhan · 2023-01-30T09:24:25Z

修复dropout1d/2d/3d中的错误，打开相关测试，修复 https://github.com/Oneflow-Inc/OneTeam/issues/1893 中的问题。

…fix_dropout_functor_error

clackhan · 2023-01-30T09:25:47Z

oneflow/core/functional/impl/nn_functor.cpp

@@ -2846,31 +2846,29 @@ Maybe<Tensor> DropoutImpl(const std::shared_ptr<one::Tensor>& input, const float
  if (p == 1) {
    std::shared_ptr<Tensor> other =
        JUST(Constant(*input->shape(), Scalar(0.0), input->dtype(), JUST(input->device())));
-    return InplaceMul(input, other);
+    return Mul(input, other);


若使用inplace版mul，则后向计算检查会报错。

clackhan · 2023-01-30T09:25:55Z

oneflow/core/functional/impl/nn_functor.cpp

  }
  std::shared_ptr<Tensor> noise = JUST(MakeFeatureNoise(input));
  noise =
      JUST(BernoulliProb(noise, 1.0 - p, noise->dtype(), JUST(one::DefaultAutoGenerator()), false));
  noise = JUST(InplaceScalarDiv(noise, Scalar(1.0 - p)));
-  noise = JUST(InplaceMul(input, noise));
-  return noise;
+  return JUST(Mul(input, noise));


clackhan · 2023-01-30T09:29:14Z

oneflow/core/functional/impl/nn_functor.cpp

+    CHECK_EQ_OR_RETURN(p < 0 || p > 1.0, false)
        << "dropout probability has to be between 0 and 1, but got " << p;


这里之前应该是写错了，结果应该是 false

clackhan · 2023-01-30T09:30:56Z

oneflow/core/functional/impl/nn_functor.cpp

           "Note that dropout1d exists to provide channel-wise dropout on inputs with 1 "
           "spatial dimension, a channel dimension, and an optional batch dimension "
           "(i.e. 2D or 3D inputs).";
    bool is_batched = (input_dim == 3);
-    std::shared_ptr<one::Tensor> result;
+    std::shared_ptr<one::Tensor> result = input;


需要先给result赋值为input，否则is_batched为true时，DropoutImpl输入为空。

clackhan · 2023-01-30T09:35:59Z

oneflow/core/functional/impl/nn_functor.cpp

+      LOG(WARNING)
+          << "dropout2d: Received a " << input_dim
+          << "-D input to dropout2d, which is deprecated "
+             "and will result in an error in a future release. To retain the behavior "
+             "and silence this warning, please use dropout instead. Note that dropout2d "
+             "exists to provide channel-wise dropout on inputs with 2 spatial dimensions, "
+             "a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).";


这里对齐pytorch，弹warning，不报错

clackhan · 2023-01-30T09:36:06Z

oneflow/core/functional/impl/nn_functor.cpp

+             "a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).";
+    }
+    if (input_dim == 3) {
+      LOG(WARNING)


clackhan · 2023-01-30T09:36:12Z

oneflow/core/functional/impl/nn_functor.cpp

-           "exists to provide channel-wise dropout on inputs with 3 spatial dimensions, "
-           "a channel dimension, and an optional batch dimension (i.e. 4D or 5D inputs).";
+    if (input_dim != 4 && input_dim != 5) {
+      LOG(WARNING)


clackhan · 2023-01-30T09:36:56Z

python/oneflow/test/modules/test_dropout.py

        device = random_device()
        x = random_tensor(ndim=random(), dim0=random(1, 8)).to(device)
-        m = torch.nn.Dropout(p=0, inplace=random_bool())
+        m = torch.nn.Dropout(p=0, inplace=False)


设置inplace为false，因为不能对 requires_grad = True 的叶子节点，做 inplace 修改

clackhan · 2023-01-30T09:38:07Z

python/oneflow/test/modules/test_dropout.py

+    @autotest(n=5, check_graph=False)
+    def test_dropout_eval(test_case):


在graph模式下（build函数中），m.eval()时非法的，故这几个用例关闭graph测试

python/oneflow/test/modules/test_dropout.py

github-actions · 2023-01-30T21:15:37Z

Static analysis with clang failed. PR label automerge has been removed

github-actions · 2023-01-31T00:48:38Z

Static analysis with clang failed. PR label automerge has been removed

github-actions · 2023-01-31T00:52:54Z

Static analysis with clang failed. PR label automerge has been removed

github-actions · 2023-01-31T05:30:44Z

CI failed when running job: cuda-module. PR label automerge has been removed

github-actions · 2023-01-31T05:31:55Z

Speed stats:

github-actions · 2023-01-31T06:56:23Z

Speed stats:

…w-Inc/oneflow into fix_dropout_functor_error

github-actions · 2023-01-31T14:52:34Z

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 142.3ms (= 14226.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 164.1ms (= 16414.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 164.1ms / 142.3ms)

OneFlow resnet50 time: 87.7ms (= 8769.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 108.9ms (= 10888.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.24 (= 108.9ms / 87.7ms)

OneFlow resnet50 time: 59.6ms (= 11911.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 79.7ms (= 15934.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 79.7ms / 59.6ms)

OneFlow resnet50 time: 46.2ms (= 9241.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 70.8ms (= 14170.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 70.8ms / 46.2ms)

OneFlow resnet50 time: 41.3ms (= 8257.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.5ms (= 15100.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.83 (= 75.5ms / 41.3ms)

github-actions · 2023-01-31T15:03:25Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9808/

clackhan added 2 commits January 30, 2023 09:22

fix_dropout_functor_error

6253909

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

f1c9822

…fix_dropout_functor_error

clackhan added enhancement automerge system labels Jan 30, 2023

clackhan requested review from hjchen2, BBuf, daquexian and jackalcooper as code owners January 30, 2023 09:24

clackhan requested review from Flowingsun007, Ldpe2G and oneflow-ci-bot and removed request for jackalcooper, daquexian and hjchen2 January 30, 2023 09:24

refine

b73661d

clackhan commented Jan 30, 2023

View reviewed changes

Ldpe2G reviewed Jan 30, 2023

View reviewed changes

python/oneflow/test/modules/test_dropout.py Show resolved Hide resolved

Ldpe2G approved these changes Jan 30, 2023

View reviewed changes

Flowingsun007 approved these changes Jan 30, 2023

View reviewed changes

Merge branch 'master' into fix_dropout_functor_error

9bf66fa

BBuf approved these changes Jan 30, 2023

View reviewed changes

mergify bot added 2 commits January 30, 2023 13:25

Merge branch 'master' into fix_dropout_functor_error

7ff9dce

Merge branch 'master' into fix_dropout_functor_error

e871e75

github-actions bot removed the automerge label Jan 30, 2023

clackhan added automerge labels Jan 31, 2023

clackhan removed the request for review from oneflow-ci-bot January 31, 2023 00:47

clackhan requested a review from oneflow-ci-bot January 31, 2023 00:47

github-actions bot removed the automerge label Jan 31, 2023

clackhan added the automerge label Jan 31, 2023

clackhan requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 31, 2023 00:51

github-actions bot removed the automerge label Jan 31, 2023

clackhan added automerge need-clean-ccache labels Jan 31, 2023

clackhan requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 31, 2023 01:46

clackhan removed the need-clean-ccache label Jan 31, 2023

github-actions bot removed the automerge label Jan 31, 2023

refine test case

e24ba70

clackhan added the automerge label Jan 31, 2023

mergify bot and others added 3 commits January 31, 2023 08:13

Merge branch 'master' into fix_dropout_functor_error

0b03dc5

refine

cde0cbf

Merge branch 'fix_dropout_functor_error' of https://github.com/Oneflo…

282ff72

…w-Inc/oneflow into fix_dropout_functor_error

mergify bot merged commit 795d19b into master Jan 31, 2023

mergify bot deleted the fix_dropout_functor_error branch January 31, 2023 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix dropout functor error #9808

Fix dropout functor error #9808

clackhan commented Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

clackhan Jan 30, 2023

github-actions bot commented Jan 30, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

		CHECK_EQ_OR_RETURN(p < 0 \|\| p > 1.0, false)
		<< "dropout probability has to be between 0 and 1, but got " << p;

		@autotest(n=5, check_graph=False)
		def test_dropout_eval(test_case):

Fix dropout functor error #9808

Fix dropout functor error #9808

Conversation

clackhan commented Jan 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 30, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023

github-actions bot commented Jan 31, 2023