operator 5.0.1 ignores CLUSTER_DOMAIN env for MINIO_ARGS env

[an-tex]

Expected Behavior

setting a custom CLUSTER_DOMAIN env when using operator 4.5.8, the MINIO_VARARGS settings correctly contain that cluster domain in the operator-webhook-secret, e.g.

MINIO_ARGS: env+tls://xx:xx.minio-operator.svc.my.domain:4222/webhook/v1/getenv/admin/minio

and i guess the operator returns the correct minio_args. the minio pods come up as intended

Current Behavior

using the new operator 5.0.1 (probably 5.0.0 affected too) with the same CLUSTER_DOMAIN env, the new sidecar creates the following config

bash-4.4$ cat /tmp/minio/config.env
export MINIO_ROOT_USER="xxx"
export MINIO_ROOT_PASSWORD="xxx"
export MINIO_BROWSER="on"
export MINIO_ARGS="http://minio-default-{0...1}.minio-hl.xxx.svc.cluster.local/export{0...1}"

hence it uses the default cluster.local domain and the minio pods fail to startup with

│ minio Error: lookup minio-default-1.minio-hl.xxx.svc.cluster.local on 10.121.0.10:53: no such host (*net.DNSError)                                                                                             │
│ minio        host="minio-default-1.minio-hl.xxx.svc.cluster.local"

Your Environment

• Version used (minio-operator): 5.0.1
• Environment name and version (e.g. kubernetes v1.17.2): v1.25.6+k0s
• Server type and version:
• Operating System and version (uname -a): Linux fsn2 5.15.0-60-generic K66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Link to your deployment file:

[dvaldivia]

Hey @an-tex perhaps after upgrade the value of CLUSTER_DOMAIN was removed from the minio-operator deployment, can you confirm your minio-operator deployment has that environment variable?

[DANic-git]

The same issue

But in pools uses cluster.local

[fgleixner]

We had the same problem, but at the end we realized, that we had a typo when upgrading the Operator using "kubectl minio init --cluster-domain ....". It seems, that if you have a typo in your cluster domain, the Operator falls back to the default "cluster.local". Initializing with the correct domain solved the problem for me.

[an-tex]

I'm using ArgoCD for the operator installation. All I'm changing is the version field in the Helm Application CRD so there shouldn't be any other differences. I've tried a few times changing it back and forth between the old and new version (and also a fresh installation), always with the same result

[DANic-git]

Any solution for the issue?

[fgleixner]

I am again having problems again with another installation. No typo today, and i cant get my minio tenants running. all pods try to connect to cluster.local. Minio Operator 5.0.2

[fgleixner]

Actually my comment 2 weeks ago was on a cluster where minio Operator 4.5.8 runs. So this comment does not affect the problem with 5.X Operator.
I think, that the sidecar and init containers introduced in 5.0.0 do not get the CLUSTER_DOMAIN env var set. So the sidecar always falls back to cluster.local.
Setting the env var for the init and sidecar container in the stateful set by hand, the tenant starts working.
I dont know if this is fixed in 5.0.3 because krew only offers 5.0.2

[fgleixner]

Ah, damn. Operator constantly deletes my changes in the STS and tenants stop working short after getting green.

[fgleixner]

Workaround until the next version of the operator is out: Scale down Operator to 0, edit STSs of tenants.