Restart job on failure for Always,OnFailure,ExitCode Policy

[yoanisgil]

What this PR does / why we need it:

There can be pod level failures caused by the system, which would previously caused the entire job to fail on all policies except ExitCode. See also #1570

Works together with kubeflow/common#195

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

#1570

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yoanisgil
To complete the pull request process, please assign gaocegege after the PR has been reviewed.
You can assign the PR to them by writing /assign @gaocegege in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnugeorge]

@yoanisgil we need to apply for other operators as well ?

/cc @gaocegege
/cc @zw0610

[coveralls]

Pull Request Test Coverage Report for Build 2464048060

• 1 of 2 (50.0%) changed or added relevant lines in 2 files are covered.
• 8 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.1%) to 36.914%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controller.v1/pytorch/pytorchjob_controller.go	0	1	0.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/mpi/mpijob_controller.go	8	77.24%

Totals
Change from base Build 2445064020:	-0.1%
Covered Lines:	2306
Relevant Lines:	6247

---

💛 - Coveralls

[gaocegege]

I think it should be added in MPIJob, too

WDYT @zw0610

[gaocegege]

Thanks for your contribution! 🎉 👍

The PR itself LGTM

[yoanisgil]

@gaocegege happy to add to MPIJob as well but I'd need some help with instructions on how to test it.

[johnugeorge]

Can you add a test case as well?

@gaocegege Why isn't this applicable to all frameworks?

[johnugeorge]

Since original PR is ready, shall we close this?

#1572

[yoanisgil]

Yup.