[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046

wanchaol · 2020-09-02T19:03:46Z

Stack from ghstack:

[c10d][jit] initial torchbind bindings for ProcessGroupNCCL #42944 [c10d][jit] initial torchbind bindings for ProcessGroup
[WIP] use lambda to register custom class init #47368 [WIP] use lambda to register custom class init
[c10d] switch ProcessGroup to be managed by intrusive_ptr #47343 [c10d] switch ProcessGroup to be managed by intrusive_ptr
[c10d] switch ProcessGroupNCCL:Options to be managed by intrusive_ptr #47075 [c10d] switch ProcessGroupNCCL:Options to be managed by intrusive_ptr
[c10d] switch Store to be managed by intrusive_ptr #47074 [c10d] switch Store to be managed by intrusive_ptr
[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046 [c10d] switch ProcessGroup::Work to be managed by intrusive_ptr

Differential Revision: D23632280

[ghstack-poisoned]

dr-ci · 2020-09-02T19:24:27Z

💊 CI failures summary and remediations

As of commit 056232f (more details on the Dr. CI page):

3/5 failures possibly* introduced in this PR
- 1/3 non-CircleCI failure(s)
2/5 broken upstream at merge base 69532c4 since Nov 10

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (1/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 11 03:37:36 ERROR [2.722s]: test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)

Nov 11 03:37:36 Traceback (most recent call last): 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper 
Nov 11 03:37:36     self._join_processes(fn) 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes 
Nov 11 03:37:36     self._check_return_codes(elapsed_time) 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes 
Nov 11 03:37:36     raise RuntimeError(error) 
Nov 11 03:37:36 RuntimeError: Processes 0 1 exited with error code 10 
Nov 11 03:37:36  
Nov 11 03:37:36 ====================================================================== 
Nov 11 03:37:36 ERROR [2.722s]: test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 
Nov 11 03:37:36 ---------------------------------------------------------------------- 
Nov 11 03:37:36 Traceback (most recent call last): 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper 
Nov 11 03:37:36     self._join_processes(fn) 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes 
Nov 11 03:37:36     self._check_return_codes(elapsed_time) 
Nov 11 03:37:36   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes 
Nov 11 03:37:36     raise RuntimeError(error) 
Nov 11 03:37:36 RuntimeError: Processes 0 1 exited with error code 10 
Nov 11 03:37:36

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (2/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 11 03:04:20 ERROR 2020-11-11T02:56:38Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" }

Nov 11 03:04:20 Traceback (most recent call last): 
Nov 11 03:04:20   File "test/run_test.py", line 867, in <module> 
Nov 11 03:04:20     main() 
Nov 11 03:04:20   File "test/run_test.py", line 850, in main 
Nov 11 03:04:20     raise RuntimeError(err_message) 
Nov 11 03:04:20 RuntimeError: distributed/test_distributed_fork failed! 
Nov 11 03:04:20 + cleanup 
Nov 11 03:04:20 + retcode=1 
Nov 11 03:04:20 + set +x 
Nov 11 03:04:20 =================== sccache compilation log =================== 
Nov 11 03:04:20 ERROR 2020-11-11T02:56:38Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Nov 11 03:04:20  
Nov 11 03:04:20 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Nov 11 03:04:20 Compile requests                    151 
Nov 11 03:04:20 Compile requests executed            60 
Nov 11 03:04:20 Cache hits                           34 
Nov 11 03:04:20 Cache hits (C/C++)                   34 
Nov 11 03:04:20 Cache misses                         25 
Nov 11 03:04:20 Cache misses (C/C++)                 25 
Nov 11 03:04:20 Cache timeouts                        0 
Nov 11 03:04:20 Cache read errors                     0

🚧 2 ongoing upstream failures:

These were probably caused by upstream breakages that are not fixed yet:

pytorch_linux_backward_compatibility_check_test since Nov 10
- 🔁 rerun
pytorch_xla_linux_bionic_py3_6_clang9_test since Nov 10
- 🔁 rerun

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.8-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 245 times.

…e_ptr" [ghstack-poisoned]

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

pritamdamania87 · 2020-10-08T01:50:53Z

torch/lib/c10d/ProcessGroup.hpp

@@ -119,33 +119,33 @@ class ProcessGroup {
    return size_;
  }

-  virtual std::shared_ptr<ProcessGroup::Work> broadcast(
+  virtual c10::intrusive_ptr<ProcessGroup::Work> broadcast(


This change is probably going to break other third party backends like: https://github.com/intel/torch-ccl/blob/master/src/ProcessGroupCCL.hpp#L136 and https://github.com/openucx/torch-ucc/blob/master/include/torch_ucc.hpp#L77.

I'm guessing this is necessary for TorchScript and there is no way around it, so should we ask the third-party libraries to make this change as well? (we can probably file issues on those repos).

cc @agolynski Since this affects the c10d extension.

Yeah agreed that we should ask them to make the changes. Do you have a list of third party backends or are these two the only two that's currently using c10d extension?

The ones I am aware of are Intel and UCX. See:

https://github.com/openucx/torch-ucc

add c10d dynamic loading mechanism and unit test #28068

Could you please check with @agolynski, he might know more context.

Hello @chengjunlu @mshiryaev @Sergei-Lebedev @srinivas212

This PR will break master -> master dependency in torch-ccl and torch-ucc.

In near future we'll be changing ProcessGroup API which will break these repos as well. Would it be okay if you depend on 1.6 (upgrade to 1.7 when released) and not on master meanwhile?

@Sergei-Lebedev @srinivas212: How does torch-ucc depend on pytorch, do you require users to install from pytorch from master branch?

Hi @agolynski @mrshenli @pritamdamania87 - after discussions with both Torch-UCC and Torch-CCL teams, the near term plan is to fix the issue in the third-party repo once this change lands.

In general, it is best we keep third-party plugins in sync w/ master.

going to land the stack soon, created openucx/torch-ucc#23 and intel/torch-ccl#11 to ucc and ccl to do the API migration.

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

facebook-github-bot · 2020-11-11T09:15:35Z

@wanchaol merged this pull request in 0650a61.

mrshenli · 2020-11-11T15:08:48Z

Hey @wanchaol, looks like this PR breaks master, shall we revert?

Nov 11 03:04:20 ======================================================================
Nov 11 03:04:20 ERROR [2.020s]: test_all_gather_multigpu_complex (__main__.TestDistBackendWithFork)
Nov 11 03:04:20 ----------------------------------------------------------------------
Nov 11 03:04:20 Traceback (most recent call last):
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper
Nov 11 03:04:20     self._join_processes(fn)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes
Nov 11 03:04:20     self._check_return_codes(elapsed_time)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes
Nov 11 03:04:20     raise RuntimeError(error)
Nov 11 03:04:20 RuntimeError: Processes 0 1 exited with error code 10
Nov 11 03:04:20 
Nov 11 03:04:20 ======================================================================
Nov 11 03:04:20 ERROR [1.920s]: test_reduce_sum_cuda (__main__.TestDistBackendWithFork)
Nov 11 03:04:20 ----------------------------------------------------------------------
Nov 11 03:04:20 Traceback (most recent call last):
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper
Nov 11 03:04:20     self._join_processes(fn)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes
Nov 11 03:04:20     self._check_return_codes(elapsed_time)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes
Nov 11 03:04:20     raise RuntimeError(error)
Nov 11 03:04:20 RuntimeError: Processes 0 1 exited with error code 10

wanchaol · 2020-11-11T18:39:35Z

Hey @wanchaol, looks like this PR breaks master, shall we revert?

Nov 11 03:04:20 ======================================================================
Nov 11 03:04:20 ERROR [2.020s]: test_all_gather_multigpu_complex (__main__.TestDistBackendWithFork)
Nov 11 03:04:20 ----------------------------------------------------------------------
Nov 11 03:04:20 Traceback (most recent call last):
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper
Nov 11 03:04:20     self._join_processes(fn)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes
Nov 11 03:04:20     self._check_return_codes(elapsed_time)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes
Nov 11 03:04:20     raise RuntimeError(error)
Nov 11 03:04:20 RuntimeError: Processes 0 1 exited with error code 10
Nov 11 03:04:20 
Nov 11 03:04:20 ======================================================================
Nov 11 03:04:20 ERROR [1.920s]: test_reduce_sum_cuda (__main__.TestDistBackendWithFork)
Nov 11 03:04:20 ----------------------------------------------------------------------
Nov 11 03:04:20 Traceback (most recent call last):
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 278, in wrapper
Nov 11 03:04:20     self._join_processes(fn)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 395, in _join_processes
Nov 11 03:04:20     self._check_return_codes(elapsed_time)
Nov 11 03:04:20   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 431, in _check_return_codes
Nov 11 03:04:20     raise RuntimeError(error)
Nov 11 03:04:20 RuntimeError: Processes 0 1 exited with error code 10

yes just reverted it and will investigate and submit it again.

…tr (#47806) Summary: Pull Request resolved: #47806 reland #44046 Test Plan: wait for ci Reviewed By: gmagogsfm Differential Revision: D24905245 fbshipit-source-id: ad75ace5432fcfd22d513878f5a73c4bb017324e

[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr

229d170

[ghstack-poisoned]

wanchaol requested review from mrshenli, pietern, pritamdamania87 and zhaojuanmao as code owners September 2, 2020 19:03

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

9f6506e

…e_ptr" [ghstack-poisoned]

wanchaol mentioned this pull request Sep 10, 2020

[c10] make intrusive_ptr available as a pybind holder type #44492

Closed

wanchaol added 3 commits September 10, 2020 14:37

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

24cc652

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

0b77de1

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

0fabd3f

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

wanchaol requested a review from rohan-varma as a code owner September 24, 2020 22:56

wanchaol added 2 commits October 2, 2020 16:38

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

50df2c1

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

b3b5479

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

wanchaol requested a review from mingzhe09088 as a code owner October 5, 2020 22:19

wanchaol added 2 commits October 5, 2020 16:02

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

e5fdc7f

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

1b75374

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

wanchaol mentioned this pull request Oct 8, 2020

[c10] fix weak_intrusive_ptr lock() #46007

Closed

pritamdamania87 reviewed Oct 8, 2020

View reviewed changes

wanchaol added 3 commits October 7, 2020 20:41

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

4045f9d

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

7a03e67

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

5309485

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

mrshenli requested a review from agolynski October 8, 2020 16:23

wanchaol added 3 commits October 8, 2020 11:16

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

639c487

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

ecd1cc6

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

a72c82e

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

gmagogsfm approved these changes Oct 21, 2020

View reviewed changes

wanchaol added 7 commits October 22, 2020 00:26

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

c9f85df

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

2029658

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

716dd6f

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

e7c5d38

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

911807b

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

9c3579a

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

c3070d3

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

This was referenced Oct 29, 2020

[c10d] switch Store to be managed by intrusive_ptr #47074

Closed

[c10d] switch ProcessGroupNCCL:Options to be managed by intrusive_ptr #47075

Closed

facebook-github-bot added the cla signed label Oct 30, 2020

wanchaol added 4 commits October 30, 2020 21:51

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

ec58283

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

534296e

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

db53f39

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

b20a3de

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

wanchaol mentioned this pull request Nov 4, 2020

[c10d] switch ProcessGroup to be managed by intrusive_ptr #47343

Closed

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

b15a6d0

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

wanchaol mentioned this pull request Nov 4, 2020

[WIP] use lambda to register custom class init #47368

Closed

wanchaol added 3 commits November 10, 2020 11:49

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

1cb14d9

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

e462a46

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

Update on "[c10d] switch ProcessGroup::Work to be managed by intrusiv…

056232f

…e_ptr" Differential Revision: [D23632280](https://our.internmc.facebook.com/intern/diff/D23632280) [ghstack-poisoned]

facebook-github-bot closed this in 0650a61 Nov 11, 2020

facebook-github-bot added the Merged label Nov 11, 2020

wanchaol mentioned this pull request Nov 11, 2020

[reland][c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #47806

Closed

facebook-github-bot deleted the gh/wanchaol/128/head branch November 14, 2020 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046

[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046

wanchaol commented Sep 2, 2020 •

edited

Loading

dr-ci bot commented Sep 2, 2020 •

edited

Loading

pritamdamania87 Oct 8, 2020

wanchaol Oct 8, 2020

mrshenli Oct 8, 2020

agolynski Oct 9, 2020

srinivas212 Oct 13, 2020

wanchaol Nov 11, 2020

facebook-github-bot commented Nov 11, 2020

mrshenli commented Nov 11, 2020

wanchaol commented Nov 11, 2020

[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046

[c10d] switch ProcessGroup::Work to be managed by intrusive_ptr #44046

Conversation

wanchaol commented Sep 2, 2020 • edited Loading

dr-ci bot commented Sep 2, 2020 • edited Loading

💊 CI failures summary and remediations

🕵️ 2 new failures recognized by patterns

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (1/2)

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (2/2)

🚧 2 ongoing upstream failures:

ci.pytorch.org: 1 failed

pritamdamania87 Oct 8, 2020

Choose a reason for hiding this comment

wanchaol Oct 8, 2020

Choose a reason for hiding this comment

mrshenli Oct 8, 2020

Choose a reason for hiding this comment

agolynski Oct 9, 2020

Choose a reason for hiding this comment

srinivas212 Oct 13, 2020

Choose a reason for hiding this comment

wanchaol Nov 11, 2020

Choose a reason for hiding this comment

facebook-github-bot commented Nov 11, 2020

mrshenli commented Nov 11, 2020

wanchaol commented Nov 11, 2020

wanchaol commented Sep 2, 2020 •

edited

Loading

dr-ci bot commented Sep 2, 2020 •

edited

Loading