deepspeed/launcher: add launcher_helper as each rank's start portal#4699
Merged
mrwyattii merged 9 commits intodeepspeedai:masterfrom Jan 27, 2024
Merged
deepspeed/launcher: add launcher_helper as each rank's start portal#4699mrwyattii merged 9 commits intodeepspeedai:masterfrom
mrwyattii merged 9 commits intodeepspeedai:masterfrom
Conversation
launcher_helper.py: init script to map env variables
Contributor
Author
|
@tjruwase Hi, could you please help to start workflow for this PR? Thanks. |
25 tasks
Contributor
Author
Collaborator
|
Hi @YizhouZ can you show the command line launched by deepspeed before and after your PR, illuatrating how your PR could help reduce command line length? Thanks! |
Contributor
Author
Sure. Without this PR, while using mpich runner, the cmd would be like When the number of total ranks goes high, the cmd would be extremely long, and trigger cmd word size limitations. After this PR, the cmd would be much shorter: |
tjruwase
requested review from
ShadenSmith and
tjruwase
and removed request for
awan-10 and
jeffra
Collaborator
|
Hi @mrwyattii Do you have any comments on this PR? This PR is essential when need to run DeepSpeed training on thousands of nodes with MPICH. The former implementation would make command line too long that overflow the command line buffer. The new implementation fix this issue. |
Contributor
Author
|
Hi @tjruwase @mrwyattii, do you have any comments on this PR? Like @delock said previously, this new implement is essential for us to enable DeepSpeed Training on a large amount of nodes. Otherwise training process would reach linux command limits. |
Contributor
mrwyattii
reviewed
Jan 25, 2024
mrwyattii
approved these changes
Jan 25, 2024
loadams
approved these changes
Jan 26, 2024
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Contributor
Author
|
Thank you so much for this quick merge! @tjruwase @mrwyattii @loadams |
mauryaavinash95
pushed a commit
to mauryaavinash95/DeepSpeed
that referenced
this pull request
Feb 17, 2024
…eepspeedai#4699) File Changes: multinode_runner.py: modify mpich runner to use launcher_helper launcher_helper.py: init script to map env variables Descriptions: Previous mpich runner would cause linux command line reaching size limitations when rank number is extremely higher. After discussion, we want to optimize it by using a helper script as each rank's start portal, which maps env variables such as rank, local_rank for deepspeed. So far we only use it for mpich runner, but it is made to be extendable, any runner could be added if facing similar situation. Only necessary args are passed to helper script. Let us know if there is any suggestion. --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Michael Wyatt <mrwyattii@gmail.com>
mauryaavinash95
pushed a commit
to mauryaavinash95/DeepSpeed
that referenced
this pull request
Feb 17, 2024
…eepspeedai#5025) Hi, for my last PR deepspeedai#4699 about launcher_helper, it mistakenly used two "PMIX". In this PR I corrected them to be "PMIX" and "PMI". And I also added _EnvironmentError_ to make sure env not get _NONE_ type, otherwise it would trigger env setting error.
rraminen
pushed a commit
to ROCm/DeepSpeed
that referenced
this pull request
May 9, 2024
…eepspeedai#4699) File Changes: multinode_runner.py: modify mpich runner to use launcher_helper launcher_helper.py: init script to map env variables Descriptions: Previous mpich runner would cause linux command line reaching size limitations when rank number is extremely higher. After discussion, we want to optimize it by using a helper script as each rank's start portal, which maps env variables such as rank, local_rank for deepspeed. So far we only use it for mpich runner, but it is made to be extendable, any runner could be added if facing similar situation. Only necessary args are passed to helper script. Let us know if there is any suggestion. --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Michael Wyatt <mrwyattii@gmail.com>
rraminen
pushed a commit
to ROCm/DeepSpeed
that referenced
this pull request
May 9, 2024
…eepspeedai#5025) Hi, for my last PR deepspeedai#4699 about launcher_helper, it mistakenly used two "PMIX". In this PR I corrected them to be "PMIX" and "PMI". And I also added _EnvironmentError_ to make sure env not get _NONE_ type, otherwise it would trigger env setting error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
File Changes:
multinode_runner.py: modify mpich runner to use launcher_helper
launcher_helper.py: init script to map env variables
Descriptions:
Previous mpich runner would cause linux command line reaching size limitations when rank number is extremely higher. After discussion, we want to optimize it by using a helper script as each rank's start portal, which maps env variables such as rank, local_rank for deepspeed. So far we only use it for mpich runner, but it is made to be extendable, any runner could be added if facing similar situation. Only necessary args are passed to helper script.
Let us know if there is any suggestion.