Recovery from "Updating Postgres server"

[ericxliu1990]

The Tenant stuck at "Updating Postgres server" after resizing the pvc of the helm chart. Here is the tenant details. Please let me know what I can provide to help with the triaging.

$ k -n tenant-tiny describe tenant
Name:         storage-tiny
Namespace:    tenant-tiny
Labels:       app=minio
              kustomize.toolkit.fluxcd.io/name=infrastructure
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  prometheus.io/path: /minio/v2/metrics/cluster
              prometheus.io/port: 9000
              prometheus.io/scrape: true
API Version:  minio.min.io/v2
Kind:         Tenant
Metadata:
  Creation Timestamp:  2023-01-08T03:30:10Z
  Generation:          33
  Managed Fields:
    API Version:  minio.min.io/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:prometheus.io/path:
          f:prometheus.io/port:
          f:prometheus.io/scrape:
        f:labels:
          f:app:
          f:kustomize.toolkit.fluxcd.io/name:
          f:kustomize.toolkit.fluxcd.io/namespace:
      f:spec:
        f:certConfig:
        f:configuration:
          f:name:
        f:env:
        f:externalCaCertSecret:
        f:externalCertSecret:
        f:externalClientCertSecrets:
        f:features:
          f:bucketDNS:
          f:domains:
        f:image:
        f:imagePullSecret:
        f:log:
          f:affinity:
            f:nodeAffinity:
            f:podAffinity:
            f:podAntiAffinity:
          f:annotations:
          f:audit:
            f:diskCapacityGB:
          f:db:
            f:affinity:
              f:nodeAffinity:
              f:podAffinity:
              f:podAntiAffinity:
            f:annotations:
            f:env:
            f:image:
            f:initimage:
            f:labels:
            f:nodeSelector:
            f:resources:
            f:securityContext:
              f:fsGroup:
              f:runAsGroup:
              f:runAsNonRoot:
              f:runAsUser:
            f:serviceAccountName:
            f:tolerations:
            f:volumeClaimTemplate:
              f:metadata:
              f:spec:
                f:accessModes:
                f:resources:
                  f:requests:
                    f:storage:
                f:storageClassName:
          f:env:
          f:image:
          f:labels:
          f:nodeSelector:
          f:resources:
          f:securityContext:
            f:fsGroup:
            f:runAsGroup:
            f:runAsNonRoot:
            f:runAsUser:
          f:serviceAccountName:
          f:tolerations:
        f:mountPath:
        f:podManagementPolicy:
        f:pools:
        f:priorityClassName:
        f:prometheus:
          f:affinity:
            f:nodeAffinity:
            f:podAffinity:
            f:podAntiAffinity:
          f:annotations:
          f:diskCapacityGB:
          f:env:
          f:image:
          f:initimage:
          f:labels:
          f:nodeSelector:
          f:resources:
          f:securityContext:
            f:fsGroup:
            f:runAsGroup:
            f:runAsNonRoot:
            f:runAsUser:
          f:serviceAccountName:
          f:sidecarimage:
          f:storageClassName:
        f:requestAutoCert:
        f:serviceAccountName:
        f:serviceMetadata:
          f:consoleServiceAnnotations:
          f:consoleServiceLabels:
          f:minioServiceAnnotations:
          f:minioServiceLabels:
        f:subPath:
        f:users:
    Manager:      kustomize-controller
    Operation:    Apply
    Time:         2023-04-30T21:25:55Z
    API Version:  minio.min.io/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:min.io/revision:
        f:labels:
          f:v1.min.io/console:
          f:v1.min.io/pool:
          f:v1.min.io/tenant:
      f:status:
        .:
        f:availableReplicas:
        f:certificates:
          .:
          f:autoCertEnabled:
        f:currentState:
        f:drivesOnline:
        f:healthMessage:
        f:healthStatus:
        f:pools:
        f:provisionedUsers:
        f:revision:
        f:syncVersion:
        f:usage:
          .:
          f:capacity:
          f:rawCapacity:
          f:rawUsage:
          f:usage:
    Manager:      minio-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-04-30T18:54:02Z
    API Version:  minio.min.io/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:scheduler:
        .:
        f:name:
      f:spec:
        f:certConfig:
          f:commonName:
          f:dnsNames:
          f:organizationName:
        f:imagePullPolicy:
        f:log:
          f:db:
            f:volumeClaimTemplate:
              f:status:
    Manager:         console
    Operation:       Update
    Time:            2023-04-30T21:03:51Z
  Resource Version:  332223348
  UID:               a3e72314-09aa-4aa5-b5ac-c35910c4ac41
Scheduler:
  Name:  
Spec:
  Cert Config:
    Common Name:  *.storage-tiny-hl.tenant-tiny.svc.cluster.local
    Dns Names:
      storage-tiny-pool-0-0.storage-tiny-hl.tenant-tiny.svc.cluster.local
    Organization Name:
      system:nodes
  Configuration:
    Name:  storage-configuration
  Env:
    Name:   MINIO_DOMAIN
    Value:  s3.ericxliu.me
    Name:   MINIO_SERVER_URL
    Value:  https://s3.ericxliu.me
    Name:   MINIO_BROWSER_REDIRECT_URL
    Value:  https://s3-console.ericxliu.me
  External Ca Cert Secret:
  External Cert Secret:
  External Client Cert Secrets:
  Features:
    Bucket DNS:  false
    Domains:
  Image:              quay.io/minio/minio:RELEASE.2023-01-02T09-40-09Z
  Image Pull Policy:  IfNotPresent
  Image Pull Secret:
  Log:
    Affinity:
      Node Affinity:
      Pod Affinity:
      Pod Anti Affinity:
    Annotations:
    Audit:
      Disk Capacity GB:  1
    Db:
      Affinity:
        Node Affinity:
        Pod Affinity:
        Pod Anti Affinity:
      Annotations:
      Env:
      Image:      
      Initimage:  
      Labels:
      Node Selector:
      Resources:
      Security Context:
        Fs Group:            999
        Run As Group:        0
        Run As Non Root:     false
        Run As User:         0
      Service Account Name:  
      Tolerations:
      Volume Claim Template:
        Metadata:
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:         20Gi
          Storage Class Name:  minio-nfs
        Status:
    Env:
    Image:  
    Labels:
    Node Selector:
    Resources:
    Security Context:
      Fs Group:            1000
      Run As Group:        0
      Run As Non Root:     false
      Run As User:         0
    Service Account Name:  
    Tolerations:
  Mount Path:             /export
  Pod Management Policy:  Parallel
  Pools:
    Name:  pool-0
    Security Context:
      Run As Group:     0
      Run As Non Root:  false
      Run As User:      0
    Servers:            1
    Volume Claim Template:
      Metadata:
        Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         100Gi
        Storage Class Name:  minio-nfs
    Volumes Per Server:      4
  Priority Class Name:       
  Prometheus:
    Affinity:
      Node Affinity:
      Pod Affinity:
      Pod Anti Affinity:
    Annotations:
    Disk Capacity GB:  1
    Env:
    Image:      
    Initimage:  
    Labels:
    Node Selector:
    Resources:
    Security Context:
      Fs Group:            1000
      Run As Group:        1000
      Run As Non Root:     true
      Run As User:         1000
    Service Account Name:  
    Sidecarimage:          
    Storage Class Name:    minio-nfs
  Request Auto Cert:       false
  Service Account Name:    
  Service Metadata:
    Console Service Annotations:
    Console Service Labels:
    Minio Service Annotations:
    Minio Service Labels:
  Sub Path:  
  Users:
    Name:  storage-user
Status:
  Available Replicas:  1
  Certificates:
    Auto Cert Enabled:  true
  Current State:        Updating Postgres server
  Drives Online:        4
  Health Message:       Service Unavailable
  Health Status:        red
  Pools:
    Legacy Security Context:  false
    Ss Name:                  storage-tiny-pool-0
    State:                    PoolInitialized
  Provisioned Users:          true
  Revision:                   0
  Sync Version:               v4.5
  Usage:
    Capacity:      37976444502016
    Raw Capacity:  429496729600
    Raw Usage:     31746739929088
    Usage:         1293505231429
Events:            <none>

[cesnietor]

We'll take a look but seems that this should be solved with the latest operator since we no longer deploy a Postgres instance.

[cniackz]

Hello team,

This was introduced back in PR: #751

It used to be: "Updating Postgres server for Log Search feature" but then changed to "Updating Postgres server". Nevertheless we no longer support this feature and the message is gone along with the PostgreSQL DB for logs.

The recommendation here is to upgrade MinIO Operator to the latest.