Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor ccl send and recv #8855

Merged
merged 51 commits into from
Aug 21, 2022
Merged

Refactor ccl send and recv #8855

merged 51 commits into from
Aug 21, 2022

Conversation

clackhan
Copy link
Contributor

@clackhan clackhan commented Aug 5, 2022

重构ccl::Send及ccl::Recv及实现，使用注册机制。

clackhan and others added 30 commits August 1, 2022 15:05
@clackhan
rename REGISTER_COLLECTIVE_COMMUNICATION_FACTORY to REGISTER_COLLECTI…
0edc3b8 
…VE_COMMUNICATION
@clackhan
refactor_ccl_allgather_and_reduce_scatter
4ea2510
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
52c8e2f 
…refactor_ccl_allgather_and_reduce_scatter
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
8ea051a 
…refactor_ccl_all_gather_and_reduce_scatter
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
043828a 
…refactor_ccl_all_gather_and_reduce_scatter
@clackhan
refactor ccl::Reduce
280a3f7
@clackhan
remove useless code
5500e9f
@clackhan
refactor ccl::Broadcast
3b3d2d9
@clackhan
fix static check error
b66da0f
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
f789a14 
…refactor_ccl_all_gather_and_reduce_scatter
@clackhan
reslove comment
ab0f0a8
@clackhan
Merge branch 'refactor_ccl_all_gather_and_reduce_scatter' of https://…
6064515 
…github.com/Oneflow-Inc/oneflow into refactor_ccl_reduce_and_broadcast
@clackhan
monir fix
31da07f
@clackhan
Merge branch 'master' into refactor_ccl_all_gather_and_reduce_scatter
ff24acc
@clackhan
reslove comments
31c157d
@clackhan
fix macro lock error
069fcd7
@clackhan
Merge branch 'refactor_ccl_all_gather_and_reduce_scatter' of https://…
0656a82 
…github.com/Oneflow-Inc/oneflow into refactor_ccl_reduce_and_broadcast
@clackhan
refine
a42f39c
@clackhan
Merge branch 'master' into refactor_ccl_all_gather_and_reduce_scatter
62bbd34
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
0b978f3 
…refactor_ccl_all_gather_and_reduce_scatter
@clackhan
Merge branch 'master' into refactor_ccl_all_gather_and_reduce_scatter
fe9d9ba
@mergify
Merge branch 'master' into refactor_ccl_all_gather_and_reduce_scatter
13b637f
@clackhan
Merge branch 'refactor_ccl_all_gather_and_reduce_scatter' of https://…
8a14774 
…github.com/Oneflow-Inc/oneflow into refactor_ccl_all_gather_and_reduce_scatter
@clackhan
fix an idiot error
944860a
@clackhan
Merge branch 'refactor_ccl_all_gather_and_reduce_scatter' of https://…
01ac174 
…github.com/Oneflow-Inc/oneflow into refactor_ccl_reduce_and_broadcast
@clackhan
Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …
b220706 
…refactor_ccl_reduce_and_broadcast
@clackhan
fix reduce functor bug
b6cefb8
@mergify
Merge branch 'master' into refactor_ccl_reduce_and_broadcast
9936f1a
@clackhan
refactor_ccl_send_and_recv
5d88ec3
@clackhan
Merge branch 'refactor_ccl_reduce_and_broadcast' of https://github.co…
392a56b 
…m/Oneflow-Inc/oneflow into refactor_ccl_send_and_recv
@github-actions
Copy link
Contributor

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8855/

@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.3ms (= 12832.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 141.8ms (= 14178.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 141.8ms / 128.3ms)

OneFlow resnet50 time: 75.4ms (= 7537.4ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.8ms (= 8479.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.12 (= 84.8ms / 75.4ms)

OneFlow resnet50 time: 48.6ms (= 9729.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 61.8ms (= 12356.1ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.27 (= 61.8ms / 48.6ms)

OneFlow resnet50 time: 36.1ms (= 7216.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 42.6ms (= 8517.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.18 (= 42.6ms / 36.1ms)

OneFlow resnet50 time: 28.2ms (= 5646.0ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.5ms (= 7493.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.33 (= 37.5ms / 28.2ms)

OneFlow swin dataloader time: 0.268s (= 53.623s / 200, num_workers=1)
PyTorch swin dataloader time: 0.148s (= 29.683s / 200, num_workers=1)
Relative speed: 0.554 (= 0.148s / 0.268s)

OneFlow swin dataloader time: 0.069s (= 13.706s / 200, num_workers=4)
PyTorch swin dataloader time: 0.040s (= 7.956s / 200, num_workers=4)
Relative speed: 0.581 (= 0.040s / 0.069s)

OneFlow swin dataloader time: 0.041s (= 8.136s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.374s / 200, num_workers=8)
Relative speed: 0.538 (= 0.022s / 0.041s)

❌ OneFlow resnet50 time: 136.7ms (= 13672.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 161.0ms (= 16097.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 161.0ms / 136.7ms)

OneFlow resnet50 time: 84.6ms (= 8464.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 103.2ms (= 10317.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 103.2ms / 84.6ms)

OneFlow resnet50 time: 58.3ms (= 11659.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.0ms (= 15596.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 78.0ms / 58.3ms)

OneFlow resnet50 time: 45.3ms (= 9068.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 70.9ms (= 14170.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.56 (= 70.9ms / 45.3ms)

OneFlow resnet50 time: 38.9ms (= 7773.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.7ms (= 13732.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.77 (= 68.7ms / 38.9ms)

@github-actions
Copy link
Contributor

CI failed when running job: cuda-misc. PR label automerge has been removed

@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.4ms (= 12840.7ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 143.0ms (= 14298.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 143.0ms / 128.4ms)

OneFlow resnet50 time: 75.4ms (= 7543.0ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 86.0ms (= 8595.7ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.14 (= 86.0ms / 75.4ms)

OneFlow resnet50 time: 48.9ms (= 9786.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 59.3ms (= 11851.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.21 (= 59.3ms / 48.9ms)

OneFlow resnet50 time: 36.4ms (= 7287.5ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 46.0ms (= 9203.6ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.26 (= 46.0ms / 36.4ms)

OneFlow resnet50 time: 28.5ms (= 5701.5ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 35.9ms (= 7173.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.26 (= 35.9ms / 28.5ms)

OneFlow swin dataloader time: 0.252s (= 50.418s / 200, num_workers=1)
PyTorch swin dataloader time: 0.148s (= 29.660s / 200, num_workers=1)
Relative speed: 0.588 (= 0.148s / 0.252s)

OneFlow swin dataloader time: 0.070s (= 13.962s / 200, num_workers=4)
PyTorch swin dataloader time: 0.040s (= 8.079s / 200, num_workers=4)
Relative speed: 0.579 (= 0.040s / 0.070s)

OneFlow swin dataloader time: 0.043s (= 8.591s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.433s / 200, num_workers=8)
Relative speed: 0.516 (= 0.022s / 0.043s)

❌ OneFlow resnet50 time: 137.0ms (= 13695.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 161.7ms (= 16174.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 161.7ms / 137.0ms)

OneFlow resnet50 time: 85.0ms (= 8505.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.5ms (= 11254.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 112.5ms / 85.0ms)

OneFlow resnet50 time: 58.4ms (= 11675.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.4ms (= 15672.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 78.4ms / 58.4ms)

OneFlow resnet50 time: 45.9ms (= 9189.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.6ms (= 13929.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.52 (= 69.6ms / 45.9ms)

OneFlow resnet50 time: 39.3ms (= 7854.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.9ms (= 13387.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.70 (= 66.9ms / 39.3ms)

@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 20, 2022 22:22
@github-actions
Copy link
Contributor

CI failed when running job: cuda-misc. PR label automerge has been removed

@github-actions
Copy link
Contributor

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8855/

@mergify mergify bot merged commit 5259a7c into master Aug 21, 2022
@mergify mergify bot deleted the refactor_ccl_send_and_recv branch August 21, 2022 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Flowingsun007 Flowingsun007 Flowingsun007 approved these changes

@hjchen2 hjchen2 hjchen2 approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
automerge enhancement refactor system
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@clackhan @hjchen2 @Flowingsun007