Support the case user down scale replicas

[Jeffwan]

Resolve issue #59

User may change replicas in a manifest update and it would be better to reconcile these changes even though not all frameworks support elastic training.

Remove resources which index is out of range in reconcile logic.

Current problem is GetPodSlices swallow these cases and ReconcilePod and ReconcileService can not get the information.

GetPodSlices logic has been changed a little bit. Instead of creating slice with size replicas. We send full snapshot back to caller. calculatePodSliceSize returns math.Max(replica. maxIndex + 1)

For example, let's assume we have pods with replica-index 0, 1, 2
If replica is 4, return a slice with size 4. [[0],[1],[2],[]], a pod with replica-index 3 will be created. (we have this support in the code because we always use replica as size of slice)

If replica is 1, return a slice with size 3. [[0],[1],[2]], pod with replica-index 1 and 2 are out of range and will be deleted. (this PR mainly focus on this part)

[kubeflow-bot]

This change is 

[Jeffwan]

One thing I am not that sure if we want to delete that pod in GetSlicesPods directly or pass to caller to handle this. To me,

1. GetSlicesPods should only build the slice rather than other logic to kubernetes resources.

2. If we delete resource directly in GetSlicesPods, that will minimize the change.

[Jeffwan]

/cc @gaocegege @richardsliu

[gaocegege]

/lgtm

[gaocegege]

One thing I am not that sure if we want to delete that pod in GetSlicesPods directly or pass to caller to handle this. To me,

1. `GetSlicesPods` should only build the slice rather than other logic to kubernetes resources.

2. If we delete resource directly in `GetSlicesPods`, that will minimize the change.

LGTM currently. I think we can open an issue to keep track of the discussion about the problem.

[johnugeorge]

Btwn, which all frameworks support this in an elastic manner?

[Jeffwan]

@johnugeorge https://github.com/pytorch/elastic allows to change number of workers dynamically.

[terrytangyuan]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [terrytangyuan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[terrytangyuan]

Thanks @Jeffwan! Feel free to open separate issues to track the remaining work.