[fix] oss dict load fix #383
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
blefaudeux
requested review from
min-xu-ai,
joshim5 and
anj-s
and removed request for
min-xu-ai
blefaudeux
commented
Feb 12, 2021
|
|
||
| # Populate the sharded optimizer state on the fly | ||
| if self.param_to_rank[param] != self.rank: | ||
| state_dict["state"][key] = None | ||
|
|
||
| if key in self.index_to_param: |
There was a problem hiding this comment.
this test was definitely wrong, I should have seen that..
blefaudeux
commented
Feb 12, 2021
| @@ -391,16 +391,16 @@ def load_state_dict(self, state_dict: Dict[str, Any]) -> None: | |||
|
|
|||
| # NOTE: PyTorch 1.5 does not index linearly but with the id(params) at saving time | |||
| # we work around that here by using the fact that the params are ordered as in the param_groups | |||
| pytorch15_index_redirect = {k: i for i, k in enumerate(state_dict["state"].keys())} | |||
There was a problem hiding this comment.
this is useless post pytorch 1.5
|
context: Planning for a follow up PR to find a unit test which exposes the previous bug (guess is that the model used was too symmetrical), but trying to fix master ASAP in the meantime |
blefaudeux
added a commit
that referenced
this pull request
Feb 12, 2021
This reverts commit 8be9d93.
blefaudeux
added a commit
that referenced
this pull request
Feb 12, 2021
facebook-github-bot
pushed a commit
to facebookresearch/fairseq
that referenced
this pull request
Feb 12, 2021
Summary: OSS removed the 'partition' key in their state dict to accommodate for changing partition size. This requires an update on the fairseq side to not look into the parameter partition, just broadcast everything, and let the optimizer on each rank decides which parameters are relevant. This diff also needs D26419095 to function completely, and blefaudeux has made fixes upstream in facebookresearch/fairscale#383 Reviewed By: myleott Differential Revision: D26382917 fbshipit-source-id: 95af1022be59e88814748acaee36a1a350f7dc5b
4 tasks
harkash
pushed a commit
to harkash/fairseq
that referenced
this pull request
Feb 23, 2021
Summary: OSS removed the 'partition' key in their state dict to accommodate for changing partition size. This requires an update on the fairseq side to not look into the parameter partition, just broadcast everything, and let the optimizer on each rank decides which parameters are relevant. This diff also needs D26419095 to function completely, and blefaudeux has made fixes upstream in facebookresearch/fairscale#383 Reviewed By: myleott Differential Revision: D26382917 fbshipit-source-id: 95af1022be59e88814748acaee36a1a350f7dc5b
jinyiyang-jhu
pushed a commit
to jinyiyang-jhu/fairseq-jyang
that referenced
this pull request
Feb 26, 2021
Summary: OSS removed the 'partition' key in their state dict to accommodate for changing partition size. This requires an update on the fairseq side to not look into the parameter partition, just broadcast everything, and let the optimizer on each rank decides which parameters are relevant. This diff also needs D26419095 to function completely, and blefaudeux has made fixes upstream in facebookresearch/fairscale#383 Reviewed By: myleott Differential Revision: D26382917 fbshipit-source-id: 95af1022be59e88814748acaee36a1a350f7dc5b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
Fix for an issue that Pyspeech was seeing, it looks like I broke the logic in the last updates of #310
The unit test did not catch that because the model was symmetrical enough, tryign to find a better test
The bug detection and fix suggestion is from Weiyi Zheng @zhengwy888
Fixes #380
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃