Skip to content

Navigation Menu

Explore
By company size
By use case
By industry
View all solutions
Topics
- AI
- DevOps
- Security
- Software Development
- View all
Explore
- GitHub Sponsors
  Fund open source developers
- The ReadME Project
  GitHub community articles
Repositories
- Enterprise platform
  AI-powered developer platform
Available add-ons
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

ray-project / ray Public

Notifications You must be signed in to change notification settings
Fork 5.8k
Star 34.4k

Code
Issues 3.7k
Pull requests 607
Actions
Projects
Security
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Security
Insights

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[air] `pyarrow.fs` persistence (9/n): `ray.train.Checkpoint` restore: Manual restore #38128

Merged

ericl merged 144 commits into ray-project:master from justinvyu:air/persistence/restore_new_checkpoint

Aug 10, 2023

Merged

[air] `pyarrow.fs` persistence (9/n): `ray.train.Checkpoint` restore: Manual restore #38128

ericl merged 144 commits into ray-project:master from justinvyu:air/persistence/restore_new_checkpoint

Aug 10, 2023

Conversation 6 Commits 144 Checks 0 Files changed

Conversation

Copy link

Contributor

justinvyu commented Aug 4, 2023 •

edited

Loading

Why are these changes needed?

This PR adds support for the new persistence path for manual trainer restoration.

API changes:

Tuner.restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)
Tuner.can_restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)

BaseTrainer.restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)
BaseTrainer.can_restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)

This is required to download the pkl files containing the info needed to manually restore an experiment, in the case that the user originally provided a custom fs.

Follow-ups needed

Tuner restore doesn't work because Tune trainables aren't supported in the new codepath yet.
- This is being tackled by [air] pyarrow.fs persistence (10/n): Unify Tune and Train sessions to support new persistence path in FunctionTrainable #38284
This PR does not yet support manual restore from a moved storage path.
- See these PRs for context:
  - [Tune] Tuner.restore from a local directory that has moved #29920
  - [Tune] Enable experiment restore from moved cloud uri #31669
- These are labeled with # TODO(justinvyu): [handle_moved_storage_path]
Update docs

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Sorry, something went wrong.

All reactions

justinvyu added 30 commits

July 27, 2023 14:24


          Pipe storage context to Trainable (used now for Trainable syncing)

abb1307

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Don't use the storage context in the trial/trainable

f6ff90a

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Disable all trainable syncing in new codepath

562369f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Pipe storage context to Train workers (not actually used yet)

95a3d20

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Update to use the new checkpoint id attribute

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Add todo comment to remove legacy path

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix race condition for setting checkpoint_uri

484e67f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix cyclical import

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add simple trainer test

8c856b8

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add legacy prefix to train session checkpoint uri

78c525f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add new checkpoint class

e97f471

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          New train session report implementation using new checkpoint

64945be

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Simplify checkpoint propagation from user code (in worker) -> trainer…

c6480c9

… -> driver

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          New tune session.report

c681ccb

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Save direction works with new checkpoint API

795bafe

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Update test with e2e trainer test

8a084bc

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Make callback supporting new checkpoint a todo for now

725d802

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Remove unnecessary comment

877acb9

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

ee4ccbd

…persistence/storage_context_to_worker_temp


          Separate out the new set checkpoint id from the old set checkpoint uri

88042b3

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

a5eeab2

…persistence/new_checkpoint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Update id -> index

a6cd9dc

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Address comments on error to raise with old ckpt type

01f34bb

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Move checkpoint upload logic to a helper fn of storage ctx

65e7a27

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint for session.py

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Drop a checkpoint marker after uploading

f2a4c36

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint for storage.py

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add a simplified checkpoint manager

49ee126

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fixes to checkpoint manager

ffa0dd4

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add unit test for simplified checkpoint manager

15553f7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Full test coverage

00cc9d7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add a simplified checkpoint manager

cb5990e

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fixes to checkpoint manager

2db9aae

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add unit test for simplified checkpoint manager

a2067b7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Remove duplicate in test

75e947c

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

justinvyu removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label

ericl reviewed

View reviewed changes

python/ray/train/base_trainer.py Outdated Show resolved Hide resolved

python/ray/tune/tuner.py Outdated Show resolved Hide resolved

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label

justinvyu added 9 commits

August 9, 2023 09:44


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

5c1c282

…persistence/restore_new_checkpoint


          read file directly from fs for trainer restore

a13d88d

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          check for existence rather than list

7e468c6

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Update tuner restore

13c224f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Mark experiment_checkpoint_dir as legacy

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Revert changes to sync down logic in trainer

0957d68

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

81c307f

…persistence/restore_new_checkpoint


          Fix lint

1716fa7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          minor fixes

5b6be43

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

justinvyu removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label

ericl approved these changes

View reviewed changes

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label

justinvyu added 2 commits

August 9, 2023 14:32


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

413ed38

…persistence/restore_new_checkpoint


          Remove backwards compatibility test

ca0a3a0

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

justinvyu added tests-ok The tagger certifies test failures are unrelated and assumes personal liability. and removed @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. labels

Copy link

Contributor Author

justinvyu commented Aug 10, 2023

@ericl Ok to merge

All reactions

Sorry, something went wrong.

ericl merged commit 982cf1a into ray-project:master

ericl mentioned this pull request

[tune/train] Implement new persistence strategy and roll out as default option #38294

Closed

justinvyu deleted the air/persistence/restore_new_checkpoint branch

August 10, 2023 17:00

shrekris-anyscale pushed a commit to shrekris-anyscale/ray that referenced this pull request


          [air] pyarrow.fs persistence (9/n): ray.train.Checkpoint restore:…

c5c313a

… Manual restore (ray-project#38128)

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

NripeshN pushed a commit to NripeshN/ray that referenced this pull request


          [air] pyarrow.fs persistence (9/n): ray.train.Checkpoint restore:…

6ff646f

… Manual restore (ray-project#38128)

Signed-off-by: NripeshN <nn2012@hw.ac.uk>

arvind-chandra pushed a commit to lmco/ray that referenced this pull request


          [air] pyarrow.fs persistence (9/n): ray.train.Checkpoint restore:…

f4aa328

… Manual restore (ray-project#38128)

Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

vymao pushed a commit to vymao/ray that referenced this pull request


          [air] pyarrow.fs persistence (9/n): ray.train.Checkpoint restore:…

d1f13ec

… Manual restore (ray-project#38128)

Signed-off-by: Victor <vctr.y.m@example.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

ericl ericl approved these changes

Assignees

Labels

The tagger certifies test failures are unrelated and assumes personal liability.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

Footer

© 2024 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Docs
Contact

You can’t perform that action at this time.