When setting restartPolicy to OnFailure in PyTorchJob, is there something like maxRetartCount

[zhiyxu]

When restartPolicy is set to OnFailure in PyTorchJob, if the worker always failed, it will be restarted continueously.
I would like to know if there is a configuration like maxRestartCount, if worker restart count reaches the limit, the PyTorchJob just fail directly and release resources.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tenzen-y]

Maybe, we can support this feature once #1718 is done.
(batch/v1 Job backoffLimitPerIndex)

/lifecycle frozen

[tenzen-y]

Currently, we can apply backOffLimit to the entire Job:

training-operator/pkg/apis/kubeflow.org/v1/common_types.go

Lines 204 to 206 in afba76b

	// Optional number of retries before marking this job failed.
	// +optional
	BackoffLimit *int32 `json:"backoffLimit,omitempty"`