Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Akoumparouli/low mem mixtral ckpt converter #8895

Merged
merged 7 commits into from
Apr 16, 2024
Merged

Akoumparouli/low mem mixtral ckpt converter #8895

merged 7 commits into from
Apr 16, 2024

Conversation

akoumpa
Copy link
Member

@akoumpa akoumpa commented Apr 12, 2024 •
edited
Loading

What does this PR do ?

This add the option --low-ram to enable checkpoint conversion in low-ram computers for large checkpoints (e.g. 250GB).

Memory reduction is achieved by not having the full converted checkpoint and the huggingface checkpoint on memory at the same time. Instead, it saves the converted checkpoint layer by layer, remove the HF checkpoint from memory and finally restores the converted layers and saves to .nemo format.

Collection: NLP

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

akoumpa added 3 commits April 11, 2024 19:51
@akoumpa
add --low-mem option to enable conversion of large checkpoints with l…
a69b8cd 
…ow ram requirements

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
delete param_to_weights
271058c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
various fixes; set hf dtype to auto
f967348 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa force-pushed the akoumparouli/low_mem_mixtral_ckpt_converter branch from f20f8d4 to f967348 Compare April 12, 2024 01:29
@akoumpa
remove unused lien
a60b1c3 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa force-pushed the akoumparouli/low_mem_mixtral_ckpt_converter branch from 1e07320 to a60b1c3 Compare April 12, 2024 16:59
@akoumpa
Copy link
Member Author

akoumpa commented Apr 12, 2024

jenkins

@akoumpa
Merge branch 'main' into akoumparouli/low_mem_mixtral_ckpt_converter
f74bf11 
Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
@akoumpa
Copy link
Member Author

akoumpa commented Apr 15, 2024

jenkins

@akoumpa akoumpa marked this pull request as ready for review April 16, 2024 16:31
@akoumpa akoumpa requested a review from yaoyu-33 April 16, 2024 16:31
@akoumpa akoumpa merged commit 468d5b6 into main Apr 16, 2024
127 checks passed
@akoumpa akoumpa deleted the akoumparouli/low_mem_mixtral_ckpt_converter branch April 16, 2024 19:39
xingyaoww pushed a commit to xingyaoww/NeMo that referenced this pull request Apr 23, 2024
@akoumpa @pre-commit-ci @xingyaoww
Akoumparouli/low mem mixtral ckpt converter (NVIDIA#8895)
c026dd5 
* add --low-mem option to enable conversion of large checkpoints with low ram requirements

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* delete param_to_weights

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* various fixes; set hf dtype to auto

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove unused lien

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
alxzhang-amazon pushed a commit to alxzhang-amazon/NeMo that referenced this pull request Apr 26, 2024
@akoumpa @pre-commit-ci @alxzhang-amazon
Akoumparouli/low mem mixtral ckpt converter (NVIDIA#8895)
2bb85b1 
* add --low-mem option to enable conversion of large checkpoints with low ram requirements

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* delete param_to_weights

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* various fixes; set hf dtype to auto

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove unused lien

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
suiyoubi pushed a commit that referenced this pull request May 2, 2024
@akoumpa @pre-commit-ci @suiyoubi
Akoumparouli/low mem mixtral ckpt converter (#8895)
79f8089 
* add --low-mem option to enable conversion of large checkpoints with low ram requirements

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* delete param_to_weights

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* various fixes; set hf dtype to auto

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove unused lien

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Ao Tang <aot@nvidia.com>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
@akoumpa @pre-commit-ci
Akoumparouli/low mem mixtral ckpt converter (NVIDIA#8895)
f7264bc 
* add --low-mem option to enable conversion of large checkpoints with low ram requirements

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* delete param_to_weights

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* various fixes; set hf dtype to auto

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove unused lien

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yaoyu-33 yaoyu-33 yaoyu-33 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@akoumpa @yaoyu-33