nccl logical kernels support nv_bfloat16

[liujuncheng]

No description provided.

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.6ms (= 12855.7ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 142.7ms (= 14273.4ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 142.7ms / 128.6ms)

OneFlow resnet50 time: 75.4ms (= 7543.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.3ms (= 8327.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.10 (= 83.3ms / 75.4ms)

OneFlow resnet50 time: 49.4ms (= 9879.0ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 61.0ms (= 12201.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.24 (= 61.0ms / 49.4ms)

OneFlow resnet50 time: 36.3ms (= 7266.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 46.5ms (= 9293.8ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.28 (= 46.5ms / 36.3ms)

OneFlow resnet50 time: 28.6ms (= 5713.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 40.0ms (= 7996.2ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.40 (= 40.0ms / 28.6ms)

OneFlow swin dataloader time: 0.263s (= 52.534s / 200, num_workers=1)
PyTorch swin dataloader time: 0.149s (= 29.851s / 200, num_workers=1)
Relative speed: 0.568 (= 0.149s / 0.263s)

OneFlow swin dataloader time: 0.109s (= 21.817s / 200, num_workers=4)
PyTorch swin dataloader time: 0.042s (= 8.335s / 200, num_workers=4)
Relative speed: 0.382 (= 0.042s / 0.109s)

OneFlow swin dataloader time: 0.039s (= 7.798s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.414s / 200, num_workers=8)
Relative speed: 0.566 (= 0.022s / 0.039s)

❌ OneFlow resnet50 time: 136.7ms (= 13665.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 166.4ms (= 16636.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 166.4ms / 136.7ms)

OneFlow resnet50 time: 84.5ms (= 8453.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 102.1ms (= 10207.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.21 (= 102.1ms / 84.5ms)

OneFlow resnet50 time: 58.0ms (= 11598.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 79.4ms (= 15876.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 79.4ms / 58.0ms)

OneFlow resnet50 time: 45.1ms (= 9028.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 70.9ms (= 14172.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.57 (= 70.9ms / 45.1ms)

OneFlow resnet50 time: 39.1ms (= 7810.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.8ms (= 13566.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.74 (= 67.8ms / 39.1ms)

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8962/