support noncontiguous binary op #9986

ofhwei · 2023-03-14T05:54:43Z

This PR now support the basic binary operation(+,-,*,/) when inputs is/are not contiguous.

for example,

import oneflow as flow

#case 1
x = flow.randn(3, 300, 400, 64).cuda()
y = flow.randn(3, 400, 300, 64).cuda()
z = x.transpose(1, 2)
ref = z / y
out = flow._C.noncontiguous_binary_op(z, y, op="div", inplace = False) 

# case 2
x = flow.randn(3, 400, 300, 64).cuda()
y = flow.randn(3, 400, 300, 64).cuda()
z = x.transpose(1, 2)
y = y.transpose(1, 2)
ref = z / y
out = flow._C.noncontiguous_binary_op(z, y, op="div", inplace = False)

The nsys results below [green box for this pr and red box for oneflow naive implementation] show that we can effectively omit the ToContiguous stages.

However, transpose the last dim is not good supported now, we will make this better soon.

CLAassistant · 2023-03-14T05:54:48Z

All committers have signed the CLA.

…dev_transpose_add

github-actions · 2023-04-25T11:50:40Z

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

…eflow into dev_transpose_add

github-actions · 2023-04-27T10:23:51Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9986/

github-actions · 2023-04-27T10:31:51Z

Speed stats:

GPU Name: NVIDIA GeForce RTX 3090 

❌ OneFlow resnet50 time: 43.2ms (= 4319.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 5756.1ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.33 (= 57.6ms / 43.2ms)

OneFlow resnet50 time: 26.3ms (= 2629.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 74.4ms (= 7438.0ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 2.83 (= 74.4ms / 26.3ms)

OneFlow resnet50 time: 56.8ms (= 11356.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 62.8ms (= 12564.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.11 (= 62.8ms / 56.8ms)

OneFlow resnet50 time: 52.8ms (= 10557.7ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 57.7ms (= 11549.3ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.09 (= 57.7ms / 52.8ms)

OneFlow resnet50 time: 44.3ms (= 8852.2ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 60.7ms (= 12134.3ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.37 (= 60.7ms / 44.3ms)

OneFlow swin dataloader time: 0.353s (= 70.550s / 200, num_workers=1)
PyTorch swin dataloader time: 0.133s (= 26.510s / 200, num_workers=1)
Relative speed: 0.376 (= 0.133s / 0.353s)

OneFlow swin dataloader time: 0.057s (= 11.495s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.624s / 200, num_workers=4)
Relative speed: 0.576 (= 0.033s / 0.057s)

OneFlow swin dataloader time: 0.030s (= 6.060s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.317s / 200, num_workers=8)
Relative speed: 0.547 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 48.6ms (= 4864.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.2ms (= 6519.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 65.2ms / 48.6ms)

OneFlow resnet50 time: 36.3ms (= 3630.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 45.4ms (= 4539.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 45.4ms / 36.3ms)

OneFlow resnet50 time: 28.4ms (= 5688.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.0ms (= 7796.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 39.0ms / 28.4ms)

OneFlow resnet50 time: 25.7ms (= 5139.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.7ms (= 7732.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.50 (= 38.7ms / 25.7ms)

OneFlow resnet50 time: 23.9ms (= 4782.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.1ms (= 7223.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.51 (= 36.1ms / 23.9ms)

github-actions · 2023-04-28T03:21:21Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9986/

github-actions · 2023-04-28T03:25:55Z

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.0ms (= 4300.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.3ms (= 5730.4ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.33 (= 57.3ms / 43.0ms)

OneFlow resnet50 time: 26.1ms (= 2607.7ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.8ms (= 3778.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.45 (= 37.8ms / 26.1ms)

OneFlow resnet50 time: 18.8ms (= 3769.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.2ms (= 7042.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.87 (= 35.2ms / 18.8ms)

OneFlow resnet50 time: 18.4ms (= 3687.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 31.6ms (= 6324.6ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.72 (= 31.6ms / 18.4ms)

OneFlow resnet50 time: 18.2ms (= 3640.7ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 27.1ms (= 5419.9ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.49 (= 27.1ms / 18.2ms)

OneFlow swin dataloader time: 0.200s (= 40.094s / 200, num_workers=1)
PyTorch swin dataloader time: 0.129s (= 25.849s / 200, num_workers=1)
Relative speed: 0.645 (= 0.129s / 0.200s)

OneFlow swin dataloader time: 0.053s (= 10.671s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.405s / 200, num_workers=4)
Relative speed: 0.600 (= 0.032s / 0.053s)

OneFlow swin dataloader time: 0.032s (= 6.334s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.307s / 200, num_workers=8)
Relative speed: 0.522 (= 0.017s / 0.032s)

❌ OneFlow resnet50 time: 47.6ms (= 4761.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 63.4ms (= 6337.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 63.4ms / 47.6ms)

OneFlow resnet50 time: 32.3ms (= 3233.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 44.8ms (= 4476.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.38 (= 44.8ms / 32.3ms)

OneFlow resnet50 time: 24.2ms (= 4836.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 41.4ms (= 8281.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.71 (= 41.4ms / 24.2ms)

OneFlow resnet50 time: 22.3ms (= 4456.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.8ms (= 7365.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.65 (= 36.8ms / 22.3ms)

OneFlow resnet50 time: 21.0ms (= 4205.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 33.9ms (= 6788.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.61 (= 33.9ms / 21.0ms)

clackhan · 2023-04-28T07:02:39Z

oneflow/user/ops/noncontiguous_binary_op.cpp

+      .Broadcast(user_op::OpArg("lhs", 0))
+      .Broadcast(user_op::OpArg("rhs", 0))
+      .Broadcast(user_op::OpArg("y", 0))


Suggested change

.Broadcast(user_op::OpArg("lhs", 0))

.Broadcast(user_op::OpArg("rhs", 0))

.Broadcast(user_op::OpArg("y", 0))

.Broadcast(user_op::OpArg("x", 0))

.Broadcast(user_op::OpArg("y", 0))

.Broadcast(user_op::OpArg("z", 0))

clackhan · 2023-04-28T07:03:19Z

oneflow/user/ops/noncontiguous_binary_op.cpp

+namespace oneflow {
+
+/*static*/ Maybe<void> NonContiguousBinaryOp::GetSbp(user_op::SbpContext* ctx) {
+  ctx->NewBuilder()


加个注释，只支持B

clackhan · 2023-04-28T07:04:15Z

oneflow/user/ops/noncontiguous_binary_op.cpp

+  ctx->SetOutputShape("y", 0, lhs);
+  const bool inplace = ctx->Attr<bool>("inplace");
+  if (inplace)
+    ctx->SetOutputStride("y", 0, ctx->InputStride("lhs", 0));


最好加个花括号

…dev_transpose_add

github-actions · 2023-05-21T17:12:31Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9986/

github-actions · 2023-05-21T17:13:33Z

Speed stats:

github-actions · 2023-05-22T02:44:56Z

Speed stats:

github-actions · 2023-05-22T03:31:58Z

Speed stats:

github-actions · 2023-05-22T13:52:40Z

Speed stats:

github-actions · 2023-05-22T14:43:39Z

Speed stats:

github-actions · 2023-05-23T16:43:08Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9986/

github-actions · 2023-05-23T16:49:28Z

Speed stats:

GPU Name: NVIDIA GeForce RTX 3090 

❌ OneFlow resnet50 time: 42.7ms (= 4270.8ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 58.3ms (= 5831.5ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.37 (= 58.3ms / 42.7ms)

OneFlow resnet50 time: 26.2ms (= 2618.2ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 36.7ms (= 3674.0ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.40 (= 36.7ms / 26.2ms)

OneFlow resnet50 time: 19.6ms (= 3917.3ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.3ms (= 7057.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.80 (= 35.3ms / 19.6ms)

OneFlow resnet50 time: 18.9ms (= 3787.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 32.8ms (= 6566.8ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.73 (= 32.8ms / 18.9ms)

OneFlow resnet50 time: 17.2ms (= 3448.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.5ms (= 5903.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.71 (= 29.5ms / 17.2ms)

OneFlow swin dataloader time: 0.200s (= 39.918s / 200, num_workers=1)
PyTorch swin dataloader time: 0.130s (= 25.948s / 200, num_workers=1)
Relative speed: 0.650 (= 0.130s / 0.200s)

OneFlow swin dataloader time: 0.055s (= 11.005s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.596s / 200, num_workers=4)
Relative speed: 0.599 (= 0.033s / 0.055s)

OneFlow swin dataloader time: 0.031s (= 6.119s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.327s / 200, num_workers=8)
Relative speed: 0.544 (= 0.017s / 0.031s)

❌ OneFlow resnet50 time: 48.6ms (= 4862.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.7ms (= 6573.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 65.7ms / 48.6ms)

OneFlow resnet50 time: 37.4ms (= 3738.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 45.9ms (= 4592.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 45.9ms / 37.4ms)

OneFlow resnet50 time: 28.8ms (= 5755.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.7ms (= 7731.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 38.7ms / 28.8ms)

OneFlow resnet50 time: 25.8ms (= 5164.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.8ms (= 7765.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.50 (= 38.8ms / 25.8ms)

OneFlow resnet50 time: 24.4ms (= 4884.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 35.8ms (= 7158.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.47 (= 35.8ms / 24.4ms)

ofhwei added 3 commits March 14, 2023 05:29

init

43785c7

add functional

b02f763

refine

440a725

ofhwei requested review from hjchen2, BBuf, jackalcooper and liujuncheng as code owners March 14, 2023 05:54

ofhwei added 6 commits March 16, 2023 01:48

update v1

ebe863c

add bwd

d433fbc

add strange comment

06254bd

rename to noncontiguous_binary_op

2f00054

Merge branch 'master' of https://github.com/OneFlow-Inc/oneflow into …

c433453

…dev_transpose_add

add unittest

1e6d46c

ofhwei requested a review from daquexian as a code owner March 17, 2023 08:49

ofhwei changed the title ~~support noncontiguous op~~ support noncontiguous binary op Mar 17, 2023

ofhwei added enhancement feature api op labels Mar 17, 2023

ofhwei and others added 7 commits March 17, 2023 09:57

set output contiguous if not inplace

a3858f0

add y_stride==1 constraint

6fe307b

add requires_grad & op check when inplace

b5fd6af

Merge branch 'master' of https://github.com/OneFlow-Inc/oneflow into …

27024c5

…dev_transpose_add

refine

93e3d8c

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

ab0869e

…dev_transpose_add

Merge branch 'master' into dev_transpose_add

3ed25bf

ofhwei requested review from clackhan and zobinHuang April 24, 2023 08:02

Merge branch 'master' into dev_transpose_add

6fd4935

ofhwei and others added 5 commits April 25, 2023 21:13

Merge branch 'master' into dev_transpose_add

3fb4635

rm unused var

be276b6

Merge branch 'dev_transpose_add' of https://github.com/Oneflow-Inc/on…

dd3bd34

…eflow into dev_transpose_add

Merge branch 'master' into dev_transpose_add

2dfe780

Merge branch 'master' into dev_transpose_add

88d4f19

Merge branch 'master' into dev_transpose_add

fbe0622

clackhan approved these changes Apr 28, 2023

View reviewed changes

ofhwei added 3 commits May 21, 2023 15:46

Merge branch 'master' of https://github.com/OneFlow-Inc/oneflow into …

0553041

…dev_transpose_add

refine

de42b2e

merge master

75ad151

ofhwei requested a review from oneflow-ci-bot May 21, 2023 16:17

ofhwei enabled auto-merge (squash) May 21, 2023 16:19

refine

7e70c3c

Merge branch 'master' into dev_transpose_add

53d94e3

Merge branch 'master' into dev_transpose_add

7ee81ff

ofhwei merged commit 08ded68 into master May 23, 2023

ofhwei deleted the dev_transpose_add branch May 23, 2023 17:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support noncontiguous binary op #9986

support noncontiguous binary op #9986

ofhwei commented Mar 14, 2023 •

edited

Loading

CLAassistant commented Mar 14, 2023 •

edited

Loading

github-actions bot commented Apr 25, 2023

github-actions bot commented Apr 27, 2023

github-actions bot commented Apr 27, 2023

github-actions bot commented Apr 28, 2023

github-actions bot commented Apr 28, 2023

clackhan Apr 28, 2023

clackhan Apr 28, 2023

clackhan Apr 28, 2023

github-actions bot commented May 21, 2023

github-actions bot commented May 21, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 23, 2023

github-actions bot commented May 23, 2023

support noncontiguous binary op #9986

support noncontiguous binary op #9986

Conversation

ofhwei commented Mar 14, 2023 • edited Loading

CLAassistant commented Mar 14, 2023 • edited Loading

github-actions bot commented Apr 25, 2023

github-actions bot commented Apr 27, 2023

github-actions bot commented Apr 27, 2023

github-actions bot commented Apr 28, 2023

github-actions bot commented Apr 28, 2023

clackhan Apr 28, 2023

Choose a reason for hiding this comment

clackhan Apr 28, 2023

Choose a reason for hiding this comment

clackhan Apr 28, 2023

Choose a reason for hiding this comment

github-actions bot commented May 21, 2023

github-actions bot commented May 21, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 22, 2023

github-actions bot commented May 23, 2023

github-actions bot commented May 23, 2023

ofhwei commented Mar 14, 2023 •

edited

Loading

CLAassistant commented Mar 14, 2023 •

edited

Loading