Universal ckp fixes #4588
Merged
Universal ckp fixes #4588
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
@stas00, FYI |
Modify ds_to_universal to remove word embeddings padding. Then, when loading from universal ckp, pad with zeros. Signed-off-by: Moshe Island <misland@habana.ai>
mosheisland
force-pushed
the
universal_ckp_fixes
branch
from
f4152a2 to
2d71d20
Compare
stas00
reviewed
Oct 31, 2023
stas00
reviewed
Oct 31, 2023
tjruwase
reviewed
Oct 31, 2023
mosheisland
force-pushed
the
universal_ckp_fixes
branch
from
0bac303 to
a661958
Compare
added 3 commits
November 1, 2023 17:40
When loading from universal checkpoint, the optimizer step is not restored. This is due to base_optimizer is not being saved in universal checkpoint. For now, set base optimizer step after it is initialized. While at it, remove unused step_count. Signed-off-by: Moshe Island <misland@habana.ai>
When loading from universal checkpoint with a different model parameter configuration, the loaded tensor parallel RNG tracker states are incorrect. In this case, we reconfigure the tensor parallel RNG tracker states with new seed values (each tp rank with a unique seed). We add an offset=iteration to the base seed. This is to ensure that when we load multiple times from universal checkpoint, we will use a different random sequence at each run. Signed-off-by: Moshe Island <misland@habana.ai>
Verify that all model patterns are matched at least once. Signed-off-by: Moshe Island <misland@habana.ai>
mosheisland
force-pushed
the
universal_ckp_fixes
branch
from
8d154d5 to
1a227aa
Compare
Contributor
Author
|
@tjruwase, I have an additional commit that adds support for universal checkpoint for llama model. Should I add it to this PR? or should I wait for this PR to be merged and create a new PR? Note that it also requires changes in Megatron-DeepSpeed PR deepspeedai/Megatron-DeepSpeed#276. Also, I suspect that the unit test failures are not due to changes in this PR, but rather some instable unit tests. I know that other PRs (not mine) are failing on same unit tests. |
Contributor
|
@mosheisland, so does that mean this PR is good to go? I was not sure because of the unresolved comments. But yes, it is okay for the llama changes to come in a different PR. No, need to delay this one any longer. |
Contributor
Author
|
@tjruwase, my understanding is that there are no open issues. can you please double check? |
tjruwase
approved these changes
Nov 9, 2023
mauryaavinash95
pushed a commit
to mauryaavinash95/DeepSpeed
that referenced
this pull request
Feb 17, 2024
Signed-off-by: Moshe Island <misland@habana.ai> Co-authored-by: Moshe Island <misland@habana.ai> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.