Avoid wrapping LightningModule in DDP plugins when not fitting #9096

four4fish · 2021-08-24T21:43:56Z

What does this PR do?

Pull changes from #8632

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

ananthsub

Thanks for carrying this forward!

pytorch_lightning/plugins/training_type/ddp.py

pytorch_lightning/plugins/training_type/ddp_spawn.py

tests/plugins/test_ddp_plugin.py

CHANGELOG.md

pytorch_lightning/plugins/training_type/ddp.py

pytorch_lightning/plugins/training_type/ddp_spawn.py

tests/plugins/test_sharded_plugin.py

codecov · 2021-08-25T21:53:09Z

Codecov Report

Merging #9096 (ca6ed38) into master (ff7305f) will decrease coverage by 4%.
The diff coverage is 67%.

@@           Coverage Diff           @@
##           master   #9096    +/-   ##
=======================================
- Coverage      92%     88%    -4%     
=======================================
  Files         176     176            
  Lines       14810   14817     +7     
=======================================
- Hits        13663   13050   -613     
- Misses       1147    1767   +620

awaelchli · 2021-08-25T22:08:08Z

@four4fish isn't this a subset of changes in #8632?

four4fish · 2021-08-25T22:16:30Z

@four4fish isn't this a subset of changes in #8632?

@awaelchli no, I have commandeered the #8632, as Ning is focus on data loader changes ~~

CHANGELOG.md

tests/plugins/test_ddp_plugin.py

awaelchli · 2021-08-25T22:21:35Z

@four4fish are we closing #8632 then? the contents are virtually identical. at least we should mark one of them as draft

four4fish · 2021-08-25T22:26:03Z

@four4fish are we closing #8632 then? the contents are virtually identical. at least we should mark one of them as draft

@awaelchli Yeah! Will close #8632. Sorry I should have marked it as commandeer PR, thanks :)!

pytorch_lightning/plugins/training_type/ddp.py

tchaton

LGTM !

CHANGELOG.md

pytorch_lightning/trainer/connectors/logger_connector/result.py

pytorch_lightning/plugins/training_type/ddp_spawn.py

CHANGELOG.md

tests/plugins/test_sharded_plugin.py

ananthsub · 2021-09-02T00:21:38Z

@four4fish - the tests are failing because the DeepSpeed plugin extends the DDP plugin. However, for DeepSpeed, we should always wrap the model, and not override the training/validation/test/predict steps (ie. DeepSpeed should continue calling self.model(*args, **args)

This is another motivation for not relying on inheritance between these plugins, as both the wrapping and checkpoint behavior differ.

tests/plugins/test_sharded_plugin.py

carmocca · 2021-09-02T03:33:36Z

CHANGELOG.md

@@ -280,6 +280,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed `DDP` "CUDA error: initialization error" due to a `copy` instead of `deepcopy` on `ResultCollection` ([#9239](https://github.com/PyTorchLightning/pytorch-lightning/pull/9239))


+- Fixed wrapping issue: avoid wrapping LightningModule with data-parallel modules when not fitting in `DDPPlugin`, `DDPSpawnPlugin`, `DDPShardedPlugin`, `DDPSpawnShardedPlugin` ([#9096]https://github.com/PyTorchLightning/pytorch-lightning/pull/9096)


The link is broken

Can be fixed with another PR - no need to open one just for this

four4fish requested review from awaelchli, Borda, carmocca, justusschock, kaushikb11, SeanNaren, tchaton and williamFalcon as code owners August 24, 2021 21:43

four4fish mentioned this pull request Aug 24, 2021

Remove reference to DistributedDataParallel from parallel plugin teardown #8943

Merged

12 tasks

ananthsub approved these changes Aug 24, 2021

View reviewed changes

pytorch_lightning/plugins/training_type/ddp.py Show resolved Hide resolved

pytorch_lightning/plugins/training_type/ddp_spawn.py Show resolved Hide resolved

tests/plugins/test_ddp_plugin.py Show resolved Hide resolved

ananthsub added distributed Generic distributed-related topic feature Is an improvement or enhancement labels Aug 24, 2021

ananthsub added this to the v1.5 milestone Aug 24, 2021

ananthsub reviewed Aug 25, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

tchaton reviewed Aug 25, 2021

View reviewed changes

pytorch_lightning/plugins/training_type/ddp.py Outdated Show resolved Hide resolved

pytorch_lightning/plugins/training_type/ddp_spawn.py Show resolved Hide resolved

awaelchli changed the title ~~Avoid wrapping LightningModule in DDP plugins when not fitting~~ Avoid wrapping LightningModule in DDP plugins when not fitting [1/2] Aug 25, 2021

four4fish changed the title ~~Avoid wrapping LightningModule in DDP plugins when not fitting [1/2]~~ Avoid wrapping LightningModule in DDP plugins when not fitting Aug 25, 2021

ananthsub reviewed Aug 25, 2021

View reviewed changes

tests/plugins/test_sharded_plugin.py Outdated Show resolved Hide resolved

four4fish force-pushed the feat/6977 branch 2 times, most recently from 870790e to 1a2245a Compare August 25, 2021 21:28

awaelchli approved these changes Aug 25, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

tests/plugins/test_ddp_plugin.py Outdated Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Aug 25, 2021

ananthsub mentioned this pull request Aug 25, 2021

Avoid wrapping LightningModule in *DataParallel overrides when not fitting #8632

Closed

12 tasks

carmocca reviewed Aug 26, 2021

View reviewed changes

pytorch_lightning/plugins/training_type/ddp.py Outdated Show resolved Hide resolved

tchaton approved these changes Aug 27, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

mergify bot added the has conflicts label Aug 27, 2021

four4fish force-pushed the feat/6977 branch from 4a7c7a4 to 8788924 Compare August 27, 2021 20:57

mergify bot added has conflicts and removed has conflicts labels Aug 27, 2021

ananthsub reviewed Aug 28, 2021

View reviewed changes

pytorch_lightning/trainer/connectors/logger_connector/result.py Outdated Show resolved Hide resolved

four4fish force-pushed the feat/6977 branch from 8788924 to 5525986 Compare August 28, 2021 22:18

mergify bot added has conflicts and removed has conflicts labels Aug 28, 2021

Borda approved these changes Aug 31, 2021

View reviewed changes

mergify bot removed the has conflicts label Aug 31, 2021

ananthsub reviewed Sep 1, 2021

View reviewed changes

pytorch_lightning/plugins/training_type/ddp_spawn.py Show resolved Hide resolved

ananthsub enabled auto-merge (squash) September 1, 2021 02:44

four4fish mentioned this pull request Sep 1, 2021

Avoid configuring SyncBatchNorm when not fitting #9243

Closed

auto-merge was automatically disabled September 1, 2021 02:55
Head branch was pushed to by a user without write access

four4fish force-pushed the feat/6977 branch from e79aafa to 17eee5e Compare September 1, 2021 02:55

ananthsub enabled auto-merge (squash) September 1, 2021 03:26

ananthsub reviewed Sep 1, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

ananthsub reviewed Sep 1, 2021

View reviewed changes

tests/plugins/test_sharded_plugin.py Outdated Show resolved Hide resolved

four4fish mentioned this pull request Sep 2, 2021

Reconciliate processes masks errors in DDP plugin if _sync_dir isn't initialized #9263

Closed

ananthsub reviewed Sep 2, 2021

View reviewed changes

tests/plugins/test_sharded_plugin.py Outdated Show resolved Hide resolved

Avoid wrapping LightningModule in DDP plugins when not fitting

91f5f0d

auto-merge was automatically disabled September 2, 2021 01:06
Head branch was pushed to by a user without write access

four4fish force-pushed the feat/6977 branch from 737223e to 91f5f0d Compare September 2, 2021 01:06

Avoid wrapping LightningModule in DDP plugins when not fitting

ca6ed38

ananthsub enabled auto-merge (squash) September 2, 2021 01:13

ananthsub merged commit a451997 into Lightning-AI:master Sep 2, 2021

carmocca reviewed Sep 2, 2021

View reviewed changes

awaelchli mentioned this pull request Oct 29, 2021

Outputs from test_step() incorrectly gathered & converted to tensors (DataParallel) #10219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid wrapping LightningModule in DDP plugins when not fitting #9096

Avoid wrapping LightningModule in DDP plugins when not fitting #9096

four4fish commented Aug 24, 2021 •

edited by ananthsub

Loading

ananthsub left a comment

codecov bot commented Aug 25, 2021 •

edited

Loading

awaelchli commented Aug 25, 2021

four4fish commented Aug 25, 2021

awaelchli commented Aug 25, 2021 •

edited

Loading

four4fish commented Aug 25, 2021

tchaton left a comment

ananthsub commented Sep 2, 2021

carmocca Sep 2, 2021

		@@ -280,6 +280,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
		- Fixed `DDP` "CUDA error: initialization error" due to a `copy` instead of `deepcopy` on `ResultCollection` ([#9239](https://github.com/PyTorchLightning/pytorch-lightning/pull/9239))


		- Fixed wrapping issue: avoid wrapping LightningModule with data-parallel modules when not fitting in `DDPPlugin`, `DDPSpawnPlugin`, `DDPShardedPlugin`, `DDPSpawnShardedPlugin` ([#9096]https://github.com/PyTorchLightning/pytorch-lightning/pull/9096)

Avoid wrapping LightningModule in DDP plugins when not fitting #9096

Avoid wrapping LightningModule in DDP plugins when not fitting #9096

Conversation

four4fish commented Aug 24, 2021 • edited by ananthsub Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

ananthsub left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 25, 2021 • edited Loading

Codecov Report

awaelchli commented Aug 25, 2021

four4fish commented Aug 25, 2021

awaelchli commented Aug 25, 2021 • edited Loading

four4fish commented Aug 25, 2021

tchaton left a comment

Choose a reason for hiding this comment

ananthsub commented Sep 2, 2021

carmocca Sep 2, 2021

Choose a reason for hiding this comment

four4fish commented Aug 24, 2021 •

edited by ananthsub

Loading

codecov bot commented Aug 25, 2021 •

edited

Loading

awaelchli commented Aug 25, 2021 •

edited

Loading