Disable istio sidecar injection in Suggestion and Training Jobs

[andreyvelich]

Fixes: kubeflow/kubeflow#4742.

I added part in Katib controller to create Suggestion deployment with sidecar.istio.io/inject: false annotation.
I am not sure that we can add this annotation to training jobs in the controller since we submit job from Unstructured object: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/trial/trial_controller.go#L271.

Because of that, I added annotations to template in each example.

/assign @johnugeorge @richardsliu

/cc @hougangliu @gaocegege @krishnadurai

---

This change is 

[johnugeorge]

Will this solve the problem without adding to the training pods? Have you tried?

Btwn, unstructured is just a map

[andreyvelich]

Will this solve the problem without adding to the training pods? Have you tried?

No, we must add it to training jobs. The main problem happens when we download datasets in training job behind istio-proxy.

Btwn, unstructured is just a map

Ok, in that case, we can work only with 3 supported job type (v1Job, TFJob and PytorchJob) and I will try to add annotation to them before submitting.

[andreyvelich]

I added few changes to this PR.

Instead of changing each example yaml file, I add istio sidecar annotation in the Katib controller.

In that case, I add annotation to the right place depends on JobKind and user can't submit any other job than Job, TFJob, PytorchJob.

If we don't want to limit user with these types of Training Jobs, we can run script with annotation only for these three types.
So, if user wants to submit other jobs, user has to add annotation manually to the yaml file.

What do you think @richardsliu @gaocegege @hougangliu @johnugeorge ?

[andreyvelich]

/retest

[andreyvelich]

/retest

[richardsliu]

/retest

[andreyvelich]

/retest

[andreyvelich]

/retest

[gaocegege]

/lgtm

[johnugeorge]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment