Support Istio-sidecar handling

[anencore94]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

I'd like to run katib-experiment with istio-sidecar injection.
I guess the reason why katib induce user to run katib-experiment with istio-sidecar injection "false" was something like this:

• katib cannot work in namespace with istio-injection enabled #955
• Support startup dependencies between containers on the same Pod kubernetes/kubernetes#65502
• Better support for sidecar containers in batch jobs istio/istio#6324
• https://medium.com/redbox-techblog/handling-istio-sidecars-in-kubernetes-jobs-c392661c4af7

Thus, the following problem is maybe the main issue.
A) katib's metrics-logger-and-collector sidecar container should know when the istio-proxy sidecar container started
B) metrics-logger-and-collector sidecar container should kill when the main container finished well or not.

---

By the way I think changing initial-logic and end-logic in metrics-collector image, katib could handle this.

Maybe something like this;

1. Handling issue B above from metrics-collector (

   katib/cmd/metricscollector/v1beta1/file-metricscollector/main.go

   Line 339 in eea6ada

   func main() {

   )

func killIstioProxySidecar() {
	klog.Infof("Let's kill istio-proxy sidecar")

	// curl -fsI -X POST http://localhost:15020/quitquitquit
	resp, err := http.Post("http://localhost:15020/quitquitquit", "text/plain", bytes.NewBufferString(""))
	if err != nil {
		// TODO 
		klog.Infof("there is no istio-proxy sidecar")
	}

	defer resp.Body.Close()
	klog.Infof("istio-proxy sidecar killed")
}

func main() {
	defer killIstioProxySidecar()

	flag.Var(&stopRules, "stop-rule", "The list of early stopping stop rules")
	flag.Parse()
	klog.Infof("Trial Name: %s", *trialName)
.....

2. Also, pass the process from istio-proxy sidecar nearby here

		// Ignore istio-sidecar processes
                // TODO this must be more sophisticated to catch istio-proxy's process
		if strings.Contains(cmdline, "envoy") || strings.Contains(cmdline, "sidecar") {
			fmt.Printf("Ignore istio-sidecar container's processes")
			continue
		}

3. Handling issue A above.
   I have no idea how to implement this now, but the primary container's command should be changed automatically by katib experiment/trial's controller as follows:

• Before

- command:
  - <YOUR_COMMAND>;

• After

- command:
  - /bin/sh
  - -c
  - |
    until curl -fsI http://localhost:15021/healthz/ready; do echo \"Waiting for Sidecar...\"; sleep 3; done;
    echo \"Sidecar available. Running the command...\";
    <YOUR_COMMAND>

• I guess there is the way since katib-controller already automatically adds echo completed > /var/log/katib/$$$$.pid at the end of primary-container's command.

---

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

• I'm afraid this feature could be generalized except "Job" kind.
  • But for "Job", this might be useful who wants to work with istio-sidecar

[anencore94]

The reason for this codes was this.

		if strings.Contains(cmdline, "envoy") || strings.Contains(cmdline, "sidecar") {
			fmt.Printf("Ignore istio-sidecar container's processes")
			continue
		}

When I created my katib-experiment with sidecar-injection == true, There was following process in view of metrics-logger-and-collector sidecar container.

[andreyvelich]

Thank you for creating this @anencore94.
Did you check this: #1383 ?
I guess waitAllProcesses flag might help us with running istio sidecar?

[anencore94]

Or I think this might be another solution:

Inject metrics-logger-and-collector container with container.lifecycle.PreStop to request quitquitquit API to istio-proxy container endpoint.

From here:

• katib/pkg/webhook/v1beta1/pod/inject_webhook.go

  Line 156 in 195db29

  injectContainer, err := s.getMetricsCollectorContainer(trial, pod)

• katib/pkg/webhook/v1beta1/pod/inject_webhook.go

  Line 216 in 195db29

  injectContainer := v1.Container{

[tenzen-y]

@anencore94 I think you are referring to a private repository.

[longpi1]

或者我认为这可能是另一种解决方案：

将带有container.lifecycle.PreStop 的metrics-logger-and-collector 容器注入到 istio-proxy 容器端点以请求quitquitquit API。

从这里：

• https://github.com/makinarocks/katib/blob/main/pkg/webhook/v1beta1/pod/inject_webhook.go#L156
• https://github.com/makinarocks/katib/blob/15ab6c9a0d8a24254e6e99e14b5377be52ec4a04/pkg/webhook/v1beta1/pod/inject_webhook.go#L217

unsee ：404

[anencore94]

Thanks @tenzen-y I changed the links

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[google-oss-prow]

@spots107: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spots107]

@anencore94 what do you think about re-opening this feature?

[anencore94]

Thanks @spots107
I guess this issue is not resolved yet.
/reopen

[google-oss-prow]

@anencore94: Reopened this issue.

In response to this:

Thanks @spots107
I guess this issue is not resolved yet.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[anencore94]

WDYT of this issue ? @kubeflow/wg-automl-leads @tenzen-y
If you think this feature is necessary, I'd like to implement

[johnugeorge]

This is definitely nice to have.

@anencore94 Thanks for following this issue

[tenzen-y]

SGTM. Thanks! @anencore94

[longpi1]

I remember I solved this problem before, if you need, you can chat with me privately, or I will propose a PR。

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tenzen-y]

/lifecycle frozen