Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restart job on failure for Always,OnFailure Policy #1572

Merged
merged 1 commit into from
Jun 9, 2022
Merged

Restart job on failure for Always,OnFailure Policy #1572

merged 1 commit into from
Jun 9, 2022

Conversation

georgkaleido
Copy link
Contributor

What this PR does / why we need it:

There can be pod level failures caused by the system, which would previously caused the entire job to fail on all policies except ExitCode. See also #1570

Works together with kubeflow/common#189

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #1570

Checklist:

  • Docs included if any changes are user facing

@aws-kf-ci-bot
Copy link
Contributor

Hi @georgkaleido. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@gaocegege
Copy link
Member

/ok-to-test

@coveralls
Copy link

coveralls commented Apr 9, 2022 •
edited
Loading

Pull Request Test Coverage Report for Build 2467493465

  • 0 of 1 (0.0%) changed or added relevant line in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.08%) to 36.978%
Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/controller.v1/pytorch/pytorchjob_controller.go 0 1 0.0%
Totals Coverage Status
Change from base Build 2445064020: -0.08%
Covered Lines: 2310
Relevant Lines: 6247
💛 - Coveralls

@zw0610
Copy link
Member

zw0610 commented May 1, 2022

LGTM
@gaocegege Do you believe we should apply this fix to all training controllers?

@gaocegege
Copy link
Member

gaocegege commented May 1, 2022 •
edited
Loading

I think so, it seems that the patch should be applied to all the operators. /cc @kubeflow/wg-training-leads

@johnugeorge
Copy link
Member

+1

@yoanisgil
Copy link

@georgkaleido do you have an image with this fix applied? I'm facing a similar issue and would love to give it a try.

@johnugeorge
Copy link
Member

Can you do a rebase?

@yoanisgil
Copy link

I've been testing this PR on preemptible GKE nodes and it does bring back failed pods from preempt events (simulated using gcloud compute instances simulate-maintenance). @johnugeorge anything I can do to get this one merged and released?

@johnugeorge
Copy link
Member

@georgkaleido Can you please rebase the PR ? We would like this get merged for next release

@yoanisgil
Copy link

@georgkaleido Can you please rebase the PR ? We would like this get merged for next release

@johnugeorge do you know when is the next release scheduled for? Just asking as I need to decide whether to wait for the next release or run this fix on my own until it's made available.

@johnugeorge
Copy link
Member

June 15th is the feature freeze and we can cut RC release within that week if things go smoothly.

If the PR author doesn't respond in this week, can you create a PR ?

@yoanisgil
Copy link

yoanisgil commented Jun 6, 2022 •
edited
Loading

June 15th is the feature freeze and we can cut RC release within that week if things go smoothly.

👍

If the PR author doesn't respond in this week, can you create a PR ?

Yup, I can do that.

@yoanisgil
Copy link

@johnugeorge if you don't mind I can take over the work done here and send a new PR? Is that ok?

@yoanisgil
Copy link

yoanisgil commented Jun 8, 2022 •
edited
Loading

@johnugeorge see #1605

@georgkaleido
Copy link
Contributor Author

Hi sry, will rebase ASAP

@georgkaleido
Restart job on failure for Always,OnFailure Policy
574efbb 
Fixes kubeflow#1570
Together with kubeflow/common#189

There can be pod level failures caused by the system, which would perviously caused the entire job to fail on all policies except ExitCode.
@georgkaleido
Copy link
Contributor Author

@johnugeorge Done

@georgkaleido
Copy link
Contributor Author

sry for the late reponse:

there is an image to test at kaleidoai/kubeflow-training-operator
master branch in https://github.com/georgkaleido/training-operator/tree/master/manifests also contains updated manifest to use this image

@johnugeorge johnugeorge mentioned this pull request Jun 9, 2022
Closed
1 task
terrytangyuan
terrytangyuan approved these changes Jun 9, 2022
View reviewed changes
Copy link
Member

@terrytangyuan terrytangyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

/lgtm
/approve

@google-oss-prow google-oss-prow bot added the lgtm label Jun 9, 2022
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: georgkaleido, terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 58cc3a0 into kubeflow:master Jun 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@terrytangyuan terrytangyuan terrytangyuan approved these changes

@jinchihe jinchihe Awaiting requested review from jinchihe

@johnugeorge johnugeorge Awaiting requested review from johnugeorge

Assignees

@terrytangyuan terrytangyuan

Labels
approved lgtm ok-to-test size/XS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

PyTorchJob: OnFailure Policy won't handle pod failure gracefully
8 participants
@georgkaleido @aws-kf-ci-bot @gaocegege @coveralls @zw0610 @johnugeorge @yoanisgil @terrytangyuan