[feature] Early Stopping Spec

[richardsliu]

/kind feature

#692

In order to make the API generic, we can use a structure similar to AlgorithmSettings (see https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1alpha3/common_types.go#L30):

type EarlyStoppingSetting struct {
	Name  string `json:"name,omitempty"`
	Value string `json:"value,omitempty"`
}

type EarlyStoppingSpec struct {
	EarlyStoppingAlgorithmName string `json:"earlyStoppingAlgorithmName,omitempty"`
        EarlyStoppingSettings []EarlyStoppingSetting `json:"earlyStoppingSettings"`
}

We can implement the Median Stopping Rule as the starting point. So the spec would look like:

algorithm:
    algorithmName: bayesianoptimization
    earlyStopping:
       earlyStoppingAlgorithmName: medianstoppingrule
       earlyStoppingSettings:
        - name: "grace_period_s"
          value: "60"    # Only stop trials that are this old
        - name: "min_samples_required"
          value: "10"    # Minimum number of previous trials required

See also: https://ray.readthedocs.io/en/latest/tune-schedulers.html#median-stopping-rule

/cc @johnugeorge @gaocegege @hougangliu

[hougangliu]

SGTM

[gaocegege]

LGTM

[johnugeorge]

LGTM

[johnugeorge]

Can you create a PR ? We can keep it in v1alpha3 as feature was not supported till now. WDYT?

[richardsliu]

v1alpha3 sounds good. I can create a PR.