Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Llama-70B-FSDP model loading issue #752

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

hlahkar
Copy link
Collaborator

@hlahkar hlahkar commented Mar 3, 2024

When running large models with FSDP, if we try to load full model params to each card we will run out of CPU RAM (refer to this these PRs huggingface/transformers#25107 huggingface/accelerate#1777. This is avoided by loading model params only to card 0 and broadcasting params to other cards from card 0. Enabled CPU_RAM_EFFICIENT_LOADING configuration in accelerate to support above described functionality.

@hlahkar hlahkar requested a review from vivekgoe March 3, 2024 09:52
@hlahkar hlahkar requested a review from regisss as a code owner March 3, 2024 09:52
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@vivekgoe vivekgoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a brief description of changes done in this PR in 1st section of PR.

@vivekgoe vivekgoe added run-test Run CI for PRs from external contributors synapse 1.15 labels Mar 4, 2024
@vivekgoe vivekgoe added run-test Run CI for PRs from external contributors and removed run-test Run CI for PRs from external contributors labels Mar 7, 2024
@vivekgoe vivekgoe merged commit 344d037 into huggingface:main Mar 7, 2024
20 of 22 checks passed
puneeshkhanna pushed a commit to puneeshkhanna/optimum-habana-fork that referenced this pull request Mar 11, 2024
HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024
Fix Llama-70B-FSDP model loading issue (#63)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-test Run CI for PRs from external contributors synapse 1.15
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants