Minio operator removing label on minio service

[manju-rn]

NOTE

I have applied a label (as required by linkerd multicluster) on minio service in cluster 1. However, looks like minio operator does not like this label and removes it (see minio operator log) after sometime - why? and how can I disable this

I0709 22:53:30.868987       1 minio-services.go:64] MinIO Services don't match: service labels don't match
I0709 22:53:30.874999       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-minipc-ns", Name:"minio", UID:"5d432c93-edfd-4462-bf71-7a069c37f45f", APIVersion:"minio.min.io/v2", ResourceVersion:"134186", FieldPath:""}): type: 'Normal' reason: 'Updated' MinIO Service Updated

The label is mirror.linkerd.io/exported=true applied to minio in ns minio-minipc-ns (created during minio tenant creation process). This label is required for linkerd to expose this service to remote cluster so that the comms between 2 clusters can happen. Below are the details of the setup

• I have 2 clusters, each installed with minio operator. Each cluster has a minio tenant.
• Clusters are connected via Linkerd mesh and multicluster communication setup. This allows minio is cluster1 to be connected to cluster 2 (remote)
• replication from minio in cluster 2 is connected to minio in cluster 1. And this works perfectly (for short duration) as long as that label on minio svc is there as linkerd uses that label to enable cross cluster comms.
  So the root cause is that the label is removed from minio, but why?

Expected Behavior

Any custom label applied in the minio svc should be retained and not removed by minio-operator

Current Behavior

Any custom label applied in the minio svc is removed by minio-operator after sometimes (ranging from 1 min- 3 mins)

Possible Solution

Option to not remove the custom labels?

Steps to Reproduce (for bugs)

1. Create a minio tenant in some workspace say minio-minipc-ns
2. Add a label to the minio svc in that namespace minio-ns. kubectl -n minio-minipc-ns label svc minio mirror.linkerd.io/exported=true
3. Watch the describe of the services and it will be removed with corresponding log in the minio-operator pod

Context

Labels are required by the Service Mesh to expose the services other cluster or even for any other general activities

Regression

Your Environment

• Version used (minio --version):
• Server setup and configuration:
• Operating System and version (uname -a):

[manju-rn]

Just to make sure this is not related to Linkerd at all. Any custom label that is applied to minio service gets removed after sometime

[jiuker]

Just to make sure this is not related to Linkerd at all. Any custom label that is applied to minio service gets removed after sometime

What's the operator version? @manju-rn

[jiuker]

fixed in #2129

[manju-rn]

@jiuker How do I get the version info? I tried exec into the minio-operator but cannot seem to find the minio command

I0710 02:44:35.841388       1 minio-services.go:64] MinIO Services don't match: service labels don't match
I0710 02:44:35.848516       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-minipc-ns", Name:"minio", UID:"5d432c93-edfd-4462-bf71-7a069c37f45f", APIVersion:"minio.min.io/v2", ResourceVersion:"146980", FieldPath:""}): type: 'Normal' reason: 'Updated' MinIO Service Updated
^C
manju@mkc:~$ k -n minio-operator exec -it  po/minio-operator-568d5455d7-89r7x -- /bin/sh
sh-5.1$ minio -version
sh: minio: command not found

[jiuker]

How do I get the version info? I tried exec into the minio-operator but cannot seem to find the minio command

I0710 02:44:35.841388       1 minio-services.go:64] MinIO Services don't match: service labels don't match
I0710 02:44:35.848516       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-minipc-ns", Name:"minio", UID:"5d432c93-edfd-4462-bf71-7a069c37f45f", APIVersion:"minio.min.io/v2", ResourceVersion:"146980", FieldPath:""}): type: 'Normal' reason: 'Updated' MinIO Service Updated
^C
manju@mkc:~$ k -n minio-operator exec -it  po/minio-operator-568d5455d7-89r7x -- /bin/sh
sh-5.1$ minio -version
sh: minio: command not found

Show the operator image @manju-rn Or restart minio-operator, log will show that

[manju-rn]

ah yes! Its 5.0.15

manju@mkc:~$ k -n minio-operator describe po/minio-operator-568d5455d7-89r7x | grep Image:
    Image:           minio/operator:v5.0.15

[harshavardhana]

fixed in #2129

closing as fixed already, rebuild from master and deploy edge or wait for the next release which is soon.

[manju-rn]

@jiuker So the v5.0.15 is already the old one? Since this one was installed like 2 days back!

[manju-rn]

@harshavardhana Can we re-open this, the issue still exists with latest version 5.0.16

manju@mkc:~$ k -n minio-operator describe po/minio-operator-74966c8f76-jzvmd | grep Image:
    Image:           minio/operator:v5.0.16

Anyway, I shifted to my last known good version 5.0.11. Strange that this issue should exist as I assume it will break lot of things, lot of people use the labels in their minio service

[jiuker]

Not v5.0.16, just rebuild from master and deploy edge or wait for the next release which is soon. @manju-rn

[manju-rn]

Not v5.0.16, just rebuild from master and deploy edge or wait for the next release which is soon. @manju-rn

Thanks for the revert. I dont't think i'll get internal approval to rebuild and deploy edge. We did have other cluster using 5.0.11 which is quite stable and hence will use that for now.

So unless i am creating a new minio tenant, the tenant creation in cluster 1 intially with 5.0.16 but now using 5.0.11 - will it have problem?