Fail to provision a Tenant in Openshift 4.13 - User Creation pending state

[maczg]

Expected Behavior

Minio tenants up&running after declaring a tenant from operator UI

Current Behavior

Tenant does not start and it is stuck on State: Provisioning initial users.

Possible Solution

It seems that the operator can't reach the MinIO tenant service when provisioning user credential. Initially, it appeared to be a network issue, but after digging deeper and examining the pool logs, even though the pods are in a running state, the logs indicate that the server is not initialized. Please, see the log attached.

Steps to Reproduce (for bugs)

1. Install minio operator from openshift OperatorHub
2. oc port-forward -n minio-operator svc/console 9443:9443
3. login and declare a tenant from UI (please, see tenant manifest below)

Context

My Openshift OKD 4.13 cluster uses a nfs storageClass provided by nfs-subdir-external-provisioner. I suspect the issue can be related to storageClass, even if it respect the requirements for PVC stated in doc.

Tenant manifest

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  creationTimestamp: "2024-04-18T08:45:33Z"
  generation: 3
  name: airflow
  namespace: airflow
  resourceVersion: "76868973"
  uid: b5008526-0b25-47b8-afb6-66a8ced071a9
scheduler:
  name: ""
spec:
  certConfig:
    commonName: '*.airflow-hl.airflow.svc.cluster.local'
    dnsNames:
    - airflow-pool-0-{0...3}.airflow-hl.airflow.svc.cluster.local
    organizationName:
    - system:nodes
  configuration:
    name: airflow-env-configuration
  credsSecret:
    name: airflow-secret
  exposeServices: {}
  features: {}
  image: minio/minio:RELEASE.2024-04-06T05-26-02Z
  imagePullPolicy: IfNotPresent
  imagePullSecret: {}
  mountPath: /export
  podManagementPolicy: Parallel
  pools:
  - name: pool-0
    resources: {}
    runtimeClassName: ""
    servers: 4
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "5368709120"
        storageClassName: nfs-client
      status: {}
    volumesPerServer: 4
  requestAutoCert: false
  serviceAccountName: airflow-sa
  users:
  - name: airflow-user-0
status:
  availableReplicas: 0
  certificates:
    autoCertEnabled: false
    customCertificates: {}
  currentState: Provisioning initial users
  healthMessage: Service Unavailable
  healthStatus: red
  pools:
  - legacySecurityContext: false
    ssName: airflow-pool-0
    state: PoolInitialized
  revision: 0
  syncVersion: v5.0.0
  usage: {}

Additional info: storage class has anyuid and privileged SCC.

Regression

Your Environment

• Version used (minio-operator): 5.0.13
• Environment name and version (e.g. kubernetes v1.17.2):
  Server Version: 4.13.0-0.okd-2023-09-30-084937
  Kubernetes Version: v1.26.4-3004+52589e6ce268bd-dirty
• Server type and version:
• Operating System and version (uname -a):
• Link to your deployment file:
  minio-pool-pod.log

[jiuker]

Delete all your minio pvc and restart pod. You reuse the pvc that been created before.

[maczg]

Thank you but I don't have any pvc/pv related to minio when i create a tenant from UI.

I already tried to restart operator deployments and create a new tenants (still, without any existing pvc related to minio pools) but pools keep failing to start

[jiuker]

Thank you but I don't have any pvc/pv related to minio when i create a tenant from UI.

I already tried to restart operator deployments and create a new tenants (still, without any existing pvc related to minio pools) but pools keep failing to start

Post the new deploy minio pod‘s log。

[maczg]

minio-pool-pod-0-0.log
minio-pool-pod-0-1.log
minio-pool-pod-0-2.log
minio-pool-pod-0-3.log

[jiuker]

API: SYSTEM.bootstrap
Time: 10:54:10 UTC 04/18/2024
Error: Detected unexpected drive ordering refusing to use the drive - poolID: 1st, found drive mounted at (set=1st, drive=8th) expected mount at (set=1st, drive=9th): /export2(b7401fd2-0f4f-4bee-9077-f5b161d40931) (*errors.errorString)
       3: internal/logger/logger.go:259:logger.LogIf()
       2: cmd/logging.go:78:cmd.bootLogIf()
       1: cmd/erasure-sets.go:464:cmd.newErasureSets.func2.1()

Can't reuse the pvc/pv/data path . @maczg

[maczg]

Thank you for your help. I'm not sure if I understood correctly. I actually didn't intentionally reuse any PVC/PV. Each tenant instance was created after cleaning up all PVCs/PVs associated with the previous one.

Is it possible that the NFS storage class is not suitable for managing PVCs/drivers for the pools?

I just tried a tenant configuration (not recommended) with 1 server in the pool and 1 drive per server, and everything went smoothly.

[jiuker]

1 server in the pool and 1 drive per server. Only one pvc.
The log reproduce steps:

minio server /data{0..3}

wait ready.
stop minio.

rename /data3 /data4
rename /data2 /data3
rename /data4 /data2

minio server /data{0..3}
Error: unexpected drive ordering on pool: 1st: found drive at (set=1st, drive=4th), expected at (set=1st, drive=3rd): /data2(): inconsistent drive found (*fmt.wrapError)

So you are reusing.
And I don't think it's good idea use nfs.