Skip to content

Conversation

@awaelchli
Copy link
Contributor

@awaelchli awaelchli commented Nov 9, 2022

What does this PR do?

Fixes #15418

The recent version of torchrun/torchelastic has a deadlock detection which kills all processes if one of them exits while others in the group perform collective operations. This means that this bypasses our own deadlock detection for which we test. So the test does no longer make sense :( However, we also have a test for the non torchelastic casek test_ddp_terminate_when_deadlock_is_detected which still covers all cases of ddp without torchelastic.

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @justusschock @awaelchli @akihironitta

@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Nov 9, 2022
@awaelchli awaelchli added tests strategy: ddp DistributedDataParallel feature Is an improvement or enhancement and removed pl Generic label for PyTorch Lightning package labels Nov 9, 2022
@awaelchli awaelchli added this to the v1.9 milestone Nov 9, 2022
@awaelchli awaelchli self-assigned this Nov 9, 2022
@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Nov 9, 2022
@awaelchli awaelchli force-pushed the tests/deadlock-detection branch from 8d897f2 to 0390c8d Compare November 9, 2022 02:19
@awaelchli awaelchli force-pushed the tests/deadlock-detection branch from d6659b8 to 34f5da8 Compare November 9, 2022 02:30
@awaelchli awaelchli changed the title WIP: Improve deadlock detection and test WIP: Update outdated deadlock detection test Nov 9, 2022
@awaelchli awaelchli changed the title WIP: Update outdated deadlock detection test Update outdated deadlock detection test Nov 9, 2022
@Borda Borda merged commit 117ea31 into master Nov 9, 2022
@Borda Borda deleted the tests/deadlock-detection branch November 9, 2022 15:17
@mergify mergify bot added the ready PRs ready to be merged label Nov 9, 2022
Borda pushed a commit that referenced this pull request Nov 10, 2022
* improve deadlock detection test
* check if failure gets caught
* remove

(cherry picked from commit 117ea31)
lexierule pushed a commit that referenced this pull request Nov 10, 2022
* improve deadlock detection test
* check if failure gets caught
* remove

(cherry picked from commit 117ea31)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Is an improvement or enhancement pl Generic label for PyTorch Lightning package ready PRs ready to be merged strategy: ddp DistributedDataParallel tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TorchElastic standalone test silently fails

3 participants