Introduce Sidecar Args

[dvaldivia]

Introduces an initContainer and a sidecar container for MinIO to retrieve the MINIO_ARGS for the tenant and to monitor the configurations secret and update it locally as soon as it changes.

Additional this sidecars will validate if root credentials are missing and if so, error out.

[cniackz]

Hello @dvaldivia I am starting to review and test this PR, I want to take my time to review this truly, so please merge PR if needed otherwise I will place my feedback soon!

[cniackz]

Cesars-MacBook-Pro:operator cniackz$ oc get serviceaccount -n tenant-lite
NAME              SECRETS   AGE
builder           1         3h4m
default           1         3h4m
deployer          1         3h4m
minio-operator    1         161m
storage-lite-sa   1         18m <------------ I can see the new service account being created

And because there is new Service Account, then we need extra line for the priviledged access in OpenShift:

oc adm policy add-scc-to-user privileged -n tenant-lite -z storage-lite-sa

Or have proper constraints for the security context, anyway, the important part is that Tenant can be deployed and initialized in OpenShift in this change:

Also as a side note, used operator-ca-tls in the meantime, since we still face TLS issue here, but will be solved in #1450

My next step is to keep exploring and test in k8s as well.

[cniackz]

Also I can see the 3 containers here and the volumes being used:

Let's see what happen if MinIO Server is terminated...

[cniackz]

It is very quickly restored, quicker than before:

[cniackz]

Great, I can reach MinIO Console Service as well via Route:

In general, this change is not breaking anything so far but just making things better and faster!!!!

[cniackz]

Great Job @dvaldivia thanks for this big Change!!! 👍

license added to files

[fgleixner]

Hi,
i just tried to understand, why the minio operator switch --cluster-domain does not work any more. All defaults to cluster.local. All tenants are broken. Is it possible, that introducing sidecars and deleting the function GetEnvHandler in http_handlers.go introduced this bug? #1535 and #1544