Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix FSDP model resume optimizer & scheduler #25852

Merged
merged 2 commits into from
Sep 1, 2023
Merged

fix FSDP model resume optimizer & scheduler #25852

merged 2 commits into from
Sep 1, 2023

Conversation

pkumc
Copy link
Contributor

@pkumc pkumc commented Aug 30, 2023

What does this PR do?

This PR is to fix FSDP model training when resume from checkpoint. All changes are made in Trainer.

  1. In save_fsdp_optimizer method, optimizer is saved with the name optimizer.bin. So I modify _load_optimizer_and_scheduler method to match it.
  2. In _save_checkpoint method, scheduler is not saved when FSDP is enabled. I think it could be a careless mistake. I refactor the code to fix it.

Who can review?

pacman100
pacman100 approved these changes Aug 30, 2023
View reviewed changes
Copy link
Contributor

@pacman100 pacman100 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @pkumc for fixing the issues with FSDP and resuming from checkpoint. This should resolved issues mentioned in #25849. As a follow up to this PR, let me add tests so that we can catch regressions going forward.

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@pacman100
Copy link
Contributor

Could you resolve the code quality issues, once that is done we are good to merge

@pkumc
Copy link
Contributor Author

pkumc commented Sep 1, 2023

Could you resolve the code quality issues, once that is done we are good to merge

@pacman100 The code quality has been resolved, thx.

@pacman100 pacman100 merged commit 024acd2 into huggingface:main Sep 1, 2023
@pkumc pkumc deleted the fix-fsdp-resume-ckpt branch September 1, 2023 10:44
parambharat pushed a commit to parambharat/transformers that referenced this pull request Sep 26, 2023
@pkumc @parambharat
fix FSDP model resume optimizer & scheduler (huggingface#25852)
92451eb 
* fix FSDP resume optimizer & scheduler

* improve trainer code quality

---------

Co-authored-by: machi04 <machi04@meituan.com>
blbadger pushed a commit to blbadger/transformers that referenced this pull request Nov 8, 2023
@pkumc @blbadger
fix FSDP model resume optimizer & scheduler (huggingface#25852)
3b2481e 
* fix FSDP resume optimizer & scheduler

* improve trainer code quality

---------

Co-authored-by: machi04 <machi04@meituan.com>
EduardoPach pushed a commit to EduardoPach/transformers that referenced this pull request Nov 18, 2023
@pkumc @EduardoPach
fix FSDP model resume optimizer & scheduler (huggingface#25852)
ddc306d 
* fix FSDP resume optimizer & scheduler

* improve trainer code quality

---------

Co-authored-by: machi04 <machi04@meituan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pacman100 pacman100 pacman100 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pkumc @HuggingFaceDocBuilderDev @pacman100