Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tenant update doesnt do anything #384

Closed
olevitt opened this issue Dec 7, 2020 · 7 comments · Fixed by #419
Closed

Tenant update doesnt do anything #384

olevitt opened this issue Dec 7, 2020 · 7 comments · Fixed by #419
Labels
bug Something isn't working

Comments

@olevitt
Copy link

olevitt commented Dec 7, 2020

Expected Behavior

When updating a tenant configuration (using either kubectl apply or kubectl edit), I expected minio-operator to process the new configuration and restart the pods.

Current Behavior

Modifying anything (tried with modifying env variables and minio image) does not do anything, even after killing the operator.

@dvaldivia
Copy link
Collaborator

Hi @olevitt, it usually takes some time to see the rolling update, we first perform a hot update of MinIO and then we perform the rolling update.

Does you kubernetes cluster has connectivity to the internet or is it restricted in any form? If we fail to download the updated MinIO image we may run into a problem, I want to confirm you are not experiencing this behavior

@dvaldivia dvaldivia added the bug Something isn't working label Dec 7, 2020
@olevitt
Copy link
Author

olevitt commented Dec 8, 2020

Hi !
Thanks for this quick reply.
The cluster's connectivity to the internet is not restricted and I have had no issues on that matter so I don't think it's related to this.

The operator logs looks more promising :

E1208 09:05:00.461257       1 main-controller.go:784] error syncing 'minio/minio': Tenant 'minio' MinIO update failed with Get "http://operator.minio-operator.svc.cluster.local:4222/webhook/v1/update/minio.RELEASE.2020-12-03T05-49-24Z.sha256sum": dial tcp: lookup operator.minio-operator.svc.cluster.local on 169.254.25.10:53: no such host
I1208 09:05:00.485451       1 main-controller.go:1406] Hit conflict issue, getting latest version of tenant
I1208 09:05:00.510915       1 main-controller.go:779] Successfully synced 'minio/minio'

And kubectl get svc -n minio-operator yields no result. There is no service corresponding to operator.minio-operator.svc.cluster.local. According to https://github.com/minio/operator/tree/master/operator-kustomize there should be one. I have no idea why there is not. Any idea how this could happen ?

When I uninstall and reinstall the operator, I still have no service created :

kubectl minio init -n minio-operator

CustomResourceDefinition tenants.minio.min.io: created
ClusterRole minio-operator-role: created
ServiceAccount minio-operator: created
ClusterRoleBinding minio-operator-binding: created
MinIO Operator Deployment minio-operator: created

I don't know how to get the version of the kubectl minio plugin I'm using but I should be using the latest as I installed it yesterday.

@harshavardhana
Copy link
Member

E1208 09:05:00.461257 1 main-controller.go:784] error syncing 'minio/minio': Tenant 'minio' MinIO update failed with Get "http://operator.minio-operator.svc.cluster.local:4222/webhook/v1/update/minio.RELEASE.2020-12-03T05-49-24Z.sha256sum": dial tcp: lookup operator.minio-operator.svc.cluster.local on 169.254.25.10:53: no such host

Looks like tenants cannot talk back to the operator here - perhaps your network policies?

@olevitt
Copy link
Author

olevitt commented Dec 8, 2020

I don't have any network policies enforced.

To me it looks like DNS resolution failure which is understandable as the service (which is linked to the operator.minio-operator.svc.cluster.local DNS name) has not been created.
So for me the root error is the fact that the service operator is not created although it should be according to https://github.com/minio/operator/blob/master/operator-kustomize/service.yaml
And for that, I have no idea why as I'm using the latest kubectl minio plugin version ...

@dvaldivia
Copy link
Collaborator

there's a log error

E1230 19:49:37.018893       1 main-controller.go:793] error syncing 'ns-4/bigdata-storage': Tar file extraction failed for file index: 3, with: EOF

I think this is the root cause, will investigate further

@sylus
Copy link

sylus commented Jan 13, 2021

Me and @zachomedia looked at this and @zachomedia wrote a patch the problem is when MinIO changed the base image from alpine to UBI and the getArtifacts call calling the largest layer no longer works to retrieve the MinIO binary files.

https://gist.githubusercontent.com/zachomedia/b98b3e7eed089ec300e18c299b8a01b1/raw/5c1a2c9b202f7688fd5b69a3a73d0195379e4896/skip.patch

@harshavardhana
Copy link
Member

@sylus feel free to send a PR

nitisht pushed a commit that referenced this issue Jan 13, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
fixes #384
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants