[train v2][doc] Add updated fault tolerance user guide #51083
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
justinvyu
requested review from
hongpeng-guo,
matthewdeng,
raulchen,
woshiyyya and
a team
as code owners
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
matthewdeng
reviewed
Mar 6, 2025
Comment on lines
5
to
6
| Handling Failures and Node Preemption (Deprecated API) | ||
| ====================================================== |
There was a problem hiding this comment.
same as other pr
Suggested change
| Handling Failures and Node Preemption (Deprecated API) | |
| ====================================================== | |
| (Deprecated) Handling Failures and Node Preemption | |
| ================================================== |
| :dedent: | ||
| :start-after: __ft_restore_from_cloud_restored_start__ | ||
| :end-before: __ft_restore_from_cloud_restored_end__ | ||
| python entrypoint.py --storage_path s3://my_bucket/ --run_name unique_run_name-id=da823d5 |
There was a problem hiding this comment.
nit: unique_run_name-id=da823d5 was kind of confusing to me, I thought this was setting a run ID at first. Maybe we can just call it unique_run_name here?
Also IDK if the user will have the wandb/mlflow run ID at this point, or if it would be generated inside the entrypoint script.
There was a problem hiding this comment.
If you want to connect to the same wandb run upon the job retry, you should set the run id on the outside and pass it in, wandb.init(..., run_id=run_id). Otherwise it's randomly generated.
I am setting a run id here as a best practice for the user submitting a job.
…n_v2/doc/fault_tolerance
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
matthewdeng
approved these changes
Mar 7, 2025
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…n_v2/doc/fault_tolerance
elimelt
pushed a commit
to elimelt/ray
that referenced
this pull request
Mar 9, 2025
…1083) `<Framework>Trainer.restore` has been deprecated. Update the fault tolerance user guide to use the new pattern of just instantiating the `<Framework>Trainer` from scratch again with the same `(storage_path, name)`. --------- Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
<Framework>Trainer.restorehas been deprecated. Update the fault tolerance user guide to use the new pattern of just instantiating the<Framework>Trainerfrom scratch again with the same(storage_path, name).