add cluster-domain to init, sidecar containers

[harshavardhana]

Closes #1535
Closes #1544

[dvaldivia]

I was about to send the same fix hahaha

[pjuarezd]

Tested using kind

setup

1. Create kind cluster with a custom domain name as example.com

kind config file

#kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: kind-cluster
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta3
  kind: ClusterConfiguration
  networking:
    dnsDomain: "example.com"
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
  - role: worker
  - role: worker

2. crate kind cluster

kind create cluster --config kind-cluster.yaml

3. Build Operator on master and load the container image in the kind cluster

TAG=minio/operator:v5.0.3-domain make docker
kind load docker-image --name kind-cluster minio/operator:v5.0.3-domain

4. Install Operator setting the custom domain to example.com

kubectl minio init --cluster-domain example.com --image minio/operator:v5.0.3-domain

verification

CLUSTER_DOMAIN environment variable will be set in minio-operator deployment in minio-operator namespace

      containers:
      - args:
        - controller
        env:
        - name: CLUSTER_DOMAIN
          value: example.com
        image: minio/operator:v5.0.3-domain 

5. Create a tenant

kubectl create namespace  myminio
kubectl minio tenant create myminio  \
		--servers             3                 \
		--volumes            6                 \
		--capacity            16Ti            \
		--storage-class   standard    \
		--namespace       myminio

Check The Statefulset myminio-ss-0 on the myminio namespace, it will have the env variable CLUSTER_DOMAIN assigned in the sidecar container.

apiVersion: apps/v1
kind: StatefulSet
...
spec:
...
      containers:
      - args:
        - sidecar
        - --tenant
        - myminio
        - --config-name
        - mitenant-env-configuration
        env:
        - name: CLUSTER_DOMAIN
          value: example.com
        image: minio/operator:v5.0.3-domain
        imagePullPolicy: IfNotPresent
        name: sidecar
...

Tenant is created correcty, see Operator logs

 k logs -n minio-operator minio-operator-7784cfc9d-dxbqq
I0421 17:28:10.268280       1 controller.go:70] Starting MinIO Operator
I0421 17:28:10.275179       1 main-controller.go:272] Setting up event handlers
I0421 17:28:10.284232       1 main-controller.go:481] Using Kubernetes CSR Version: v1
I0421 17:28:10.284252       1 main-controller.go:501] STS Api server is not enabled, not starting
I0421 17:28:10.284301       1 leaderelection.go:248] attempting to acquire leader lease minio-operator/minio-operator-lock...
I0421 17:28:10.287095       1 leaderelection.go:258] successfully acquired lease minio-operator/minio-operator-lock
I0421 17:28:10.287168       1 main-controller.go:530] minio-operator-7784cfc9d-dxbqq: I am the leader, applying leader labels on myself
I0421 17:28:10.287370       1 main-controller.go:385] Waiting for Upgrade Server to start
I0421 17:28:10.287377       1 main-controller.go:389] Starting Tenant controller
I0421 17:28:10.287379       1 main-controller.go:392] Waiting for informer caches to sync
I0421 17:28:10.287385       1 main-controller.go:397] Starting workers
I0421 17:28:10.287391       1 main-controller.go:425] Console TLS is not enabled
I0421 17:28:10.287472       1 main-controller.go:347] Starting HTTP Upgrade Tenant Image server
I0421 17:32:34.724012       1 minio.go:274] Generating private key
I0421 17:32:34.724200       1 minio.go:287] Generating CSR with CN=*.myminio-hl.myminio.svc.cluster.local
I0421 17:32:34.731873       1 csr.go:182] Start polling for certificate of csr/myminio-myminio-csr, every 5s, timeout after 20m0s
I0421 17:32:34.731919       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2641", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created
I0421 17:32:39.741954       1 csr.go:208] Certificate successfully fetched, creating secret with Private key and Certificate
E0421 17:32:39.747975       1 main-controller.go:666] error syncing 'myminio/myminio': waiting for minio cert
I0421 17:32:44.748120       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2667", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created
I0421 17:32:44.774778       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0421 17:32:44.811345       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2673", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created
I0421 17:32:44.822947       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0421 17:32:44.839995       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2679", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created
I0421 17:32:44.926362       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2679", FieldPath:""}): type: 'Normal' reason: 'SACreated' Service Account Created
I0421 17:32:44.930935       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2679", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created
I0421 17:32:44.935508       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2679", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created
I0421 17:32:45.727123       1 operator.go:115] Checking if this is OpenShift Environment to append the certificates...
I0421 17:32:45.727166       1 operator.go:119] csr-signer secret wasn't found, very likely this is not OpenShift but k8s Vanilla or other...
I0421 17:32:46.138160       1 main-controller.go:905] Detected we are updating a legacy tenant deployment
I0421 17:32:46.143895       1 main-controller.go:944] 'myminio/myminio': Deploying pool ss-0
I0421 17:32:46.158332       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2695", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool ss-0 created
I0421 17:32:46.737115       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:47.127738       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
E0421 17:32:47.376273       1 main-controller.go:666] error syncing 'myminio/myminio': Put "https://minio.myminio.svc.example.com/minio/admin/v3/add-user?accessKey=TLE7UK0OWNK31NHAWGH5": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:47.376296       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2698", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: Put "https://minio.myminio.svc.example.com/minio/admin/v3/add-user?accessKey=TLE7UK0OWNK31NHAWGH5": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:47.526352       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:53.230411       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:53.239152       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:54.210035       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:54.214969       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:54.878306       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:55.846784       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:58.227509       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:58.234214       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:32:59.890794       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
I0421 17:33:06.916372       1 monitoring.go:124] 'myminio/myminio' Failed to get cluster health: Get "https://minio.myminio.svc.example.com/minio/health/cluster": dial tcp 10.96.251.152:443: connect: connection refused
E0421 17:34:04.795640       1 main-controller.go:666] error syncing 'myminio/myminio': context deadline exceeded
I0421 17:34:04.795994       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"2963", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: context deadline exceeded
I0421 17:34:25.350055       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0421 17:34:44.944728       1 event.go:285] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"myminio", Name:"myminio", UID:"99c61c4d-ab8c-4b80-b6f5-d1543e746b30", APIVersion:"minio.min.io/v2", ResourceVersion:"3139", FieldPath:""}): type: 'Normal' reason: 'UsersCreated' Users created
I0421 17:34:44.947299       1 status.go:55] Hit conflict issue, getting latest version of tenant

Tenant status is initialized:

❯ kubectl get tenant -n myminio
NAME      STATE         AGE
myminio   Initialized   10m

all tenant pods are running, see a tenant pod log:

Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 138
All MinIO sub-systems initialized successfully in 6.848583ms
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-03-24T21-41-23Z (go1.19.7 linux/arm64)
Status: 6 Online, 0 Offline.
API: https://minio.myminio.svc.example.com
Console: https://10.244.1.5:9443 https://127.0.0.1:9443
Documentation: https://min.io/docs/minio/linux/index.html
You are running an older version of MinIO released 3 weeks ago
Update: Run `mc admin update`