Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support model size calculation for FSDP & DeepSpeed (stage-3) #10291

Open
rohitgr7 opened this issue Nov 1, 2021 · 0 comments
Open

Support model size calculation for FSDP & DeepSpeed (stage-3) #10291

rohitgr7 opened this issue Nov 1, 2021 · 0 comments
Assignees
@awaelchli
Labels
feature Is an improvement or enhancement strategy: deepspeed strategy: fsdp Fully Sharded Data Parallel

Comments

@rohitgr7
Copy link
Contributor

rohitgr7 commented Nov 1, 2021 •
edited by github-actions bot
Loading

🚀 Feature

Currently LightningModule.model_size (deprecated) relies on:
https://github.com/PyTorchLightning/pytorch-lightning/blob/45c45dc7b018f9a2db60f5df1a3f7dbbb45ccb36/pytorch_lightning/utilities/memory.py#L165-L179

to calculate model size. Since we are going to keep get_model_size_mb utility function, we should have a way to calculate model_sizes for sharded models.

cc @Borda @awaelchli @carmocca @SeanNaren @rohitgr7 @akihironitta
@rohitgr7 rohitgr7 added the feature Is an improvement or enhancement label Nov 1, 2021
@rohitgr7 rohitgr7 mentioned this issue Feb 28, 2022
Closed
@SeanNaren SeanNaren mentioned this issue May 5, 2022
Closed
@SeanNaren SeanNaren mentioned this issue Jun 28, 2022
Merged
12 tasks
@awaelchli awaelchli added strategy: fsdp Fully Sharded Data Parallel and removed strategy: fairscale fsdp (removed) Fully Sharded Data Parallel labels Mar 18, 2023
@awaelchli awaelchli assigned awaelchli and unassigned SeanNaren Mar 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@awaelchli awaelchli

Labels
feature Is an improvement or enhancement strategy: deepspeed strategy: fsdp Fully Sharded Data Parallel
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@awaelchli @SeanNaren @rohitgr7