Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Unified Checkpoint] update non-merge checkpoint loading, move async_save_info.json location #9321

Merged
merged 25 commits into from
Oct 28, 2024
Merged

[Unified Checkpoint] update non-merge checkpoint loading, move async_save_info.json location #9321

merged 25 commits into from
Oct 28, 2024

Conversation

DesmonDay
Copy link
Contributor

@DesmonDay DesmonDay commented Oct 28, 2024 •
edited
Loading

PR types

Others

PR changes

Others

Description

  1. Update non-merge checkpoint loading to multi-threading load.
  2. Update async_save_info.json location.

DesmonDay and others added 23 commits August 21, 2024 14:56
@DesmonDay
[Unified checkpoint] update optimizer async save signal
5451d31
@gongel
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
4f0b61a 
…nto develop
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
68470aa 
…nto develop
@DesmonDay
update paddlepaddle
15e83e2
@DesmonDay
split param
6837b2f
@DesmonDay
add save for split param
633d742
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
55186d7 
…nto add_split_param
@DesmonDay
fix save split_param
b6aa309
@DesmonDay
add load uc split_param
bf5d72b
@DesmonDay
update uc files
9fdaae2
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
9a210db 
…nto add_split_param
@DesmonDay
update uc files
19071ef
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
223e089 
…nto add_split_param
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
ae9ddce 
…nto add_split_param
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
4ab0df1 
…nto add_split_param
@DesmonDay
update split_param loading
cbbc074
@DesmonDay
mkdir unified_checkpoint directory
7678fad
@DesmonDay
rename file
238888d
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
780040e 
…nto add_split_param
@DesmonDay
update async handler
b219ba6
@DesmonDay
update files
dbd13df
@DesmonDay
update async_save_info.json file place
c758d96
@DesmonDay
Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…
e6db62e 
…nto add_split_param
@paddle-bot Paddle-bot
Copy link

paddle-bot bot commented Oct 28, 2024

Thanks for your contribution!

@DesmonDay DesmonDay changed the title [Unified Checkpoint] [Unified Checkpoint] Update async_save_info.json location. Oct 28, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 28, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 20.00000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 52.86%. Comparing base (c9d5673) to head (fd5dea0).
Report is 257 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/trainer/trainer.py 20.00% 4 Missing ⚠️
...p/trainer/unified_checkpoint/unified_checkpoint.py 20.00% 4 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff            @@
##           develop    #9321   +/-   ##
========================================
  Coverage    52.86%   52.86%           
========================================
  Files          669      669           
  Lines       107240   107229   -11     
========================================
- Hits         56688    56686    -2     
+ Misses       50552    50543    -9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DesmonDay DesmonDay changed the title [Unified Checkpoint] Update async_save_info.json location. [Unified Checkpoint] update non-merge checkpoint loading, move async_save_info.json location Oct 28, 2024
ZHUI
ZHUI approved these changes Oct 28, 2024
View reviewed changes
Copy link
Collaborator

@ZHUI ZHUI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ZHUI ZHUI merged commit ce3a1ce into PaddlePaddle:develop Oct 28, 2024
9 of 12 checks passed
DesmonDay added a commit to DesmonDay/PaddleNLP that referenced this pull request Oct 28, 2024
@DesmonDay
[Unified Checkpoint] update non-merge checkpoint loading, move async_…
9c941b3 
…save_info.json location (PaddlePaddle#9321)

* [Unified checkpoint] update optimizer async save signal

* update async_save_info.json file place
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ZHUI ZHUI ZHUI approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@DesmonDay @ZHUI @gongel