Provisioned Tenant from Operator Istio sidecar issues

[joelcomp1]

If I have istio sidecars on for both Minio Operator and Tenant objects, whent he Tenant pods get created the validate-arguments init container fails because the mTLS tunnel isn't up yet. This maybe more of a feature request then a bug (or just documentation) can the validate-arguments init container be disabled or re-configured when istio sidecars are injected?

Expected Behavior

Tenant pods should come up with Istio enabled

Current Behavior

validate-arguments crash loop because they can't connect to the pod

Possible Solution

Disable init containers on Tenant deployments when istio sidecars are injected (manually via Tenant or Operator config)

Steps to Reproduce (for bugs)

1. Install minio operator with namespace labeled as istio-injection=enabled
2. Install a tenant CRD in namespace with istio-injection=enabled

Context

Trying to run Minio with Istio mTLS

Regression

No

Your Environment

• Version used (minio-operator): 5.0.12
• Environment name and version (e.g. kubernetes v1.17.2): K8s RKE2 1.29.1
• Server type and version: Rocky 8.9
• Operating System and version (uname -a): Linux 6.5.0-18-generic Ability to set custom MINIO_STORAGE_CLASS_STANDARD environment variable in the container #18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 11:40:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[fouadsemaan]

Since you are running operator and tenant on Istio, do you still have pre-packaged minio tls enabled? Did you turn it off? Also do you have authorization policy set to allow operator namespace to link to tenant namespace?

[joelcomp1]

I do still have it enabled but per all these old issues that fixed service labels I assumed that was OK: #749

The issue really is the way the init containers work with the service mesh due to the the proxy not being up to deal with the traffic. I can't tell from the tenant if the init container is required or of it could optionally be disabled. Maybe I just have to deal with this until SidecarContainers feature gate is GA in K8s

[DanSalt]

We're also observing the same thing as we try to upgrade to the latest major version of Minio.

I guess fundamentally the question is - is Operator-based Minio designed to work with a Service Mesh like Istio?

Our initial attempts were around using the "special" Istio user for the InitContainer. But that causes issues in that the configuration file is then owned by the wrong user, which causes issues later when running the sidecar.

Just curious how other Service Mesh users are working around this.

[ramondeklein]

What is preventing MinIO from running inside the service mesh? From what I read in this topic, I understand that the initialization pods are scheduled before the Istio sidecar is injected and that causes the pod to fail. If you can share logging that shows what is actually failing, then that may help.

This is the first topic I've seen in months where a service mesh is used (and doesn't work), so demand for it is low. Therefor, we probably won't officially support it. If we can implement a simple fix to make it work, then please submit a PR. It's not that we don't want to support it, but resources are limited.