Fix Never Resume Policy for Experiment

[andreyvelich]

I fixed some problems in Never resume policy.

1. IsCompletedExperimentRestartable should return true only if ResumePolicy: LongRunning.
2. Call terminateSuggestion only for ResumePolicy: Never.
3. Reconcile can call terminateSuggestion twice. I added check if Suggestion is already succeeded.
4. I believe we should call g.Status().Update() instead of g.Update() to update status of suggestion CR.
5. terminateSuggestion just returns error, I don't think we need to requeue Reconcile: reconcile.Result{Requeue: true}.
6. Fix some problems with Logging.
7. Add e2e test for Never Resume Experiment.

/assign @johnugeorge @gaocegege
/cc @sperlingxx

[kubeflow-bot]

This change is 

[gaocegege]

/lgtm

[gaocegege]

/retest

[sperlingxx]

Sry for making so many bugs, and I am so working on this ^ ^.

[sperlingxx]

I think this paragraph could be modified to something below

	if instance.IsCompleted() {
		needRestart := false
		// Check if completed instance is restartable
		// Experiment is restartable only if it is in succeeded state by reaching max trials
		if util.IsCompletedExperimentRestartable(instance) {
			// Check if max trials is reconfigured
			if (instance.Spec.MaxTrialCount != nil &&
				*instance.Spec.MaxTrialCount != instance.Status.Trials) ||
				(instance.Spec.MaxTrialCount == nil && instance.Status.Trials != 0) {
				msg := "Experiment is restarted"
				instance.MarkExperimentStatusRestarting(util.ExperimentRestartingReason, msg)
				needRestart = true
			}
		}
		// If experiment who doesn't need to restart is completed without running trials, stop reconcile
		if !needRestart && !instance.HasRunningTrials() {
			if instance.Spec.ResumePolicy != experimentsv1alpha3.LongRunning {
				return r.terminateSuggestion(instance)
			}
			return reconcile.Result{}, nil
		}
	}

Otherwise, terminateSuggestion won't work when corresponding experiment has been completed for reaching max trials.

[andreyvelich]

@sperlingxx Why terminateSuggestion won't run when experiment finishes with reaching max trials?

I believe, Reconcile loop runs with Succeeded experiment state, after reaching this step: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/experiment/util/status_util.go#L182.

I am wondering, do we actually need to check if Experiment has running trials in the controller @gaocegege @johnugeorge :
https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/experiment/experiment_controller.go#L209 ?
Or we can just return reconcile.Result{}, nil without this check if Experiment is Completed?

[sperlingxx]

@andreyvelich I think util.IsCompletedExperimentRestartable will be true when experiment finishes with reaching max trials. But, for now, terminateSuggestion will only be called when util.IsCompletedExperimentRestartable is false.

[andreyvelich]

Yeah, but util.IsCompletedExperimentRestartable returns True only if ResumePolicy: LongRunning.

https://github.com/kubeflow/katib/pull/1184/files#diff-f0faa0b63a35acff95fe5e9d93d594ffR201

[sperlingxx]

@andreyvelich Oh, I got it!

[gaocegege]

@sperlingxx Everyone creates bugs. Do not need to worry about it.

[andreyvelich]

@sperlingxx No worries! Thank you for the review.

[andreyvelich]

@sperlingxx Why terminateSuggestion won't run when experiment finishes with reaching max trials?

I believe, Reconcile loop runs with Succeeded experiment state, after reaching this step: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/experiment/util/status_util.go#L182.

I am wondering, do we actually need to check if Experiment has running trials in the controller @gaocegege @johnugeorge :
https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/experiment/experiment_controller.go#L209 ?
Or we can just return reconcile.Result{}, nil without this check if Experiment is Completed?

[andreyvelich]

/retest

[andreyvelich]

/retest

[andreyvelich]

I think it was a problem with new version of Scikit Learn 0.23.
I created PR to change version to 0.22 in this PR: #1187.
I reverted this commit from that PR.

[sperlingxx]

/approve

[sperlingxx]

/lgtm

[andreyvelich]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, sperlingxx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment