fix UpdateJobStatusInApiServer when gang-scheduling is enabled

[zw0610]

When --enable-gang-scheduling is enabled, for jobs with PodGroup in pending phase, the operator will update the JobStatus with only its LastReconcileTime being changed. That means the jobStatus.replicaStatues could be nil when being pushed to the APIServer, triggering an error message.

This PR add a fix procedure for all controllers to give an empty value for jobStatus.replicaStatues if nil.

[coveralls]

Pull Request Test Coverage Report for Build 1996436561

• 5 of 15 (33.33%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.008%) to 36.815%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controller.v1/mpi/mpijob_controller.go	1	3	33.33%
pkg/controller.v1/pytorch/pytorchjob_controller.go	1	3	33.33%
pkg/controller.v1/mxnet/mxjob_controller.go	0	3	0.0%
pkg/controller.v1/xgboost/xgboostjob_controller.go	0	3	0.0%

Totals
Change from base Build 1972516378:	-0.008%
Covered Lines:	2277
Relevant Lines:	6185

---

💛 - Coveralls

[gaocegege]

/lgtm

[Jeffwan]

That's a nice catch.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege, zw0610

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gaocegege,zw0610]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jeffwan]

I have one comment on this. Since we don't use webhook, it's a little bit hard to mutate obj prior reconciliation.

We do have similar cases like below. The tricky thing is we should invoke these method in the right place. In this cases, it definitely operate object before initialization which bring the problems.

training-operator/pkg/controller.v1/tensorflow/util.go

Line 62 in 4b5d651

func setRestartPolicy(podTemplateSpec *corev1.PodTemplateSpec, spec *commonv1.ReplicaSpec) {

training-operator/pkg/controller.v1/tensorflow/util.go

Lines 99 to 105 in 4b5d651

	func initializeReplicaStatuses(jobStatus *commonv1.JobStatus, rtype commonv1.ReplicaType) {
	if jobStatus.ReplicaStatuses == nil {
	jobStatus.ReplicaStatuses = make(map[commonv1.ReplicaType]*commonv1.ReplicaStatus)
	}
	jobStatus.ReplicaStatuses[rtype] = &commonv1.ReplicaStatus{}
	}

[gaocegege]

Should we involve a webhook for the operator?

[johnugeorge]

Isn't this to be handled by a webhook?