GroupedMatmulBiasOp add tensor parallel sbp

[liujuncheng]

No description provided.

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

❌ OneFlow resnet50 time: 141.2ms (= 14124.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 144.1ms (= 14414.8ms / 100, input_shape=[16, 3, 224, 224])
❌ Relative speed: 1.02 (= 144.1ms / 141.2ms)

OneFlow resnet50 time: 83.3ms (= 8328.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 88.2ms (= 8823.1ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.06 (= 88.2ms / 83.3ms)

OneFlow resnet50 time: 51.0ms (= 10191.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 56.3ms (= 11264.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.11 (= 56.3ms / 51.0ms)

OneFlow resnet50 time: 33.6ms (= 6717.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 45.0ms (= 9001.1ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.34 (= 45.0ms / 33.6ms)

OneFlow resnet50 time: 25.4ms (= 5081.4ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 40.0ms (= 8007.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.58 (= 40.0ms / 25.4ms)

OneFlow swin dataloader time: 0.241s (= 48.104s / 200, num_workers=1)
PyTorch swin dataloader time: 0.153s (= 30.576s / 200, num_workers=1)
Relative speed: 0.636 (= 0.153s / 0.241s)

OneFlow swin dataloader time: 0.071s (= 14.255s / 200, num_workers=4)
PyTorch swin dataloader time: 0.041s (= 8.197s / 200, num_workers=4)
Relative speed: 0.575 (= 0.041s / 0.071s)

OneFlow swin dataloader time: 0.039s (= 7.784s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.470s / 200, num_workers=8)
Relative speed: 0.574 (= 0.022s / 0.039s)

❌ OneFlow resnet50 time: 154.2ms (= 15422.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 166.3ms (= 16632.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.08 (= 166.3ms / 154.2ms)

OneFlow resnet50 time: 93.9ms (= 9389.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 103.9ms (= 10385.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.11 (= 103.9ms / 93.9ms)

OneFlow resnet50 time: 60.8ms (= 12167.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 80.4ms (= 16071.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 80.4ms / 60.8ms)

OneFlow resnet50 time: 44.0ms (= 8793.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.4ms (= 13870.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.58 (= 69.4ms / 44.0ms)

OneFlow resnet50 time: 36.9ms (= 7381.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.5ms (= 13703.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.86 (= 68.5ms / 36.9ms)

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9934/