Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[sharded plugin] Fix check for fp16 precision #7825

Merged
merged 57 commits into from
Jun 4, 2021
Merged

[sharded plugin] Fix check for fp16 precision #7825

Changes from 51 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
89f284d
Fix some test errors
Mar 23, 2021
80cfbff
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 23, 2021
536c132
checkpoint consolidation
Mar 24, 2021
f172101
Update ddp_spawn.py
shuyingsunshine21 Mar 24, 2021
bf70e43
Update test_metric_result_integration.py
shuyingsunshine21 Mar 24, 2021
ea74906
Update test_results.py
shuyingsunshine21 Mar 24, 2021
a9aae99
Update utils.py
shuyingsunshine21 Mar 24, 2021
70fe5da
Update utils.py
shuyingsunshine21 Mar 24, 2021
0d23d75
Update test_all_gather_grad.py
shuyingsunshine21 Mar 24, 2021
ca6f98b
Update test_all_gather_grad.py
shuyingsunshine21 Mar 24, 2021
c5053da
Merge pull request #1 from shuyingsunshine21/shuyingsunshine21-checkp…
shuyingsunshine21 Mar 24, 2021
9d4a2b8
Update test_results.py
shuyingsunshine21 Mar 24, 2021
7635b4f
Revert "Update test_results.py"
shuyingsunshine21 Mar 24, 2021
d64f90c
Revert "Merge pull request #1 from shuyingsunshine21/shuyingsunshine2…
shuyingsunshine21 Mar 24, 2021
dcdcd29
Revert "Update test_all_gather_grad.py"
shuyingsunshine21 Mar 24, 2021
8651d54
Revert "Update utils.py"
shuyingsunshine21 Mar 24, 2021
15f4b9e
Revert "Update utils.py"
shuyingsunshine21 Mar 24, 2021
250d0aa
Revert "Update test_results.py"
shuyingsunshine21 Mar 24, 2021
6c095b2
Revert "Update test_metric_result_integration.py"
shuyingsunshine21 Mar 24, 2021
8222dc9
Revert "Update ddp_spawn.py"
shuyingsunshine21 Mar 24, 2021
3a9fde9
Revert "checkpoint consolidation"
shuyingsunshine21 Mar 24, 2021
7a369f4
Revert "Revert "checkpoint consolidation""
shuyingsunshine21 Mar 24, 2021
b4a0b9e
Revert "Revert "Revert "checkpoint consolidation"""
shuyingsunshine21 Mar 24, 2021
5cf1db1
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 24, 2021
0ce7e05
Revert "Revert "Update ddp_spawn.py""
shuyingsunshine21 Mar 24, 2021
fe9736d
Revert "Revert "Update test_metric_result_integration.py""
shuyingsunshine21 Mar 24, 2021
c314ef6
Revert "Revert "Update test_results.py""
shuyingsunshine21 Mar 24, 2021
c3feda0
Revert "Revert "Update utils.py""
shuyingsunshine21 Mar 24, 2021
c759477
Revert "Revert "Update test_all_gather_grad.py""
shuyingsunshine21 Mar 24, 2021
7a8e540
Merge branch 'master' of https://github.com/shuyingsunshine21/pytorch…
Mar 24, 2021
ab8b849
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 24, 2021
4e67db2
modify distributed environment to make test pass
Mar 24, 2021
67b6188
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Mar 25, 2021
179d47e
rebase
Apr 8, 2021
f9afa07
rebase to upstream master
Apr 8, 2021
b461e44
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Apr 8, 2021
e1bbc4d
fix version for ddp plugin test
Apr 8, 2021
8270d0d
fix
Apr 8, 2021
803d5dd
fix
Apr 8, 2021
ce1a19b
changelog
Apr 8, 2021
c6a13be
Update CHANGELOG.md
carmocca Apr 9, 2021
e274758
Merge pull request #3 from shuyingsunshine21/ddp_plugin_test_fix
shuyingsunshine21 Apr 10, 2021
f337156
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Apr 14, 2021
35bb931
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Apr 15, 2021
c938a9c
rebase
May 11, 2021
ad93cde
rebase
May 11, 2021
8b5eca8
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
May 15, 2021
1467a9f
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
May 21, 2021
99669c0
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
May 28, 2021
843e52c
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
Jun 3, 2021
c55b35f
fix precision for DDPShardedPlugin
Jun 3, 2021
749e277
comments
Jun 3, 2021
6597fec
add changlog
Jun 3, 2021
7d628cb
Update pytorch_lightning/plugins/training_type/sharded.py
ananthsub Jun 4, 2021
f8138aa
comment
Jun 4, 2021
a82266e
Merge branch 'fix_precision_bug' of https://github.com/shuyingsunshin…
Jun 4, 2021
07a6207
rebase to resolve conflict
Jun 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pytorch_lightning/plugins/training_type/sharded.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def _reinit_optimizers_with_oss(self):
optim_class = type(optimizer)
zero_optimizer = OSS(params=optimizer.param_groups, optim=optim_class, **optimizer.defaults)
if _FAIRSCALE_OSS_FP16_BROADCAST_AVAILABLE:
is_fp16 = self.lightning_module.trainer.precision == 16
is_fp16 = self.lightning_module.trainer.precision == "mixed"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
is_fp16 = self.lightning_module.trainer.precision == "mixed"
precision = self.lightning_module.trainer.precision
is_fp16 = precision == "mixed" or precision == 16

just in case we enable true fp16 later

# For multi-node training, compressing the model shards in fp16 before broadcasting
# improves performance. When using PyTorch AMP, it will not degrade
# the model performance.
Expand Down