[bitnami/minio] Access Denied when Deploying Bitnami/MinIO 2021.3.26-debian-10-r1 #5951

ethernoy · 2021-03-30T02:37:59Z

Which chart:
MinIO ( 6.7.1 )

Describe the bug
Encountered access denied error when deploying image Bitnami/MinIO 2021.3.26-debian-10-r1 using distributed mode. MinIO container restarts after this error occurs.

To Reproduce
Steps to reproduce the behavior:

Deploy Bitnami/MinIO 2021.3.26-debian-10-r1 using distributed mode.

Expected behavior
MinIO works normally

Version of Helm and Kubernetes:

Output of helm version:

version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5+vmware.1", GitCommit:"1abde2b816bac0da89c6c71360799c681094ca0e", GitTreeState:"clean", BuildDate:"2020-06-29T22:31:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Additional context
The kernel I am running MinIO on is 4.19.129-1.ph3-esx. Admission controller is enabled and MinIO is granted root permission.

The text was updated successfully, but these errors were encountered:

marcosbc · 2021-03-30T08:38:20Z

Hi @ethernoy, could you share more details on how you're deploying Minio? It works for me.

$ helm install myminio bitnami/minio --set mode=distributed

$ kubectl get pods
...
myminio-0                                  1/1     Running            0          8m27s
myminio-1                                  1/1     Running            0          8m27s
myminio-2                                  1/1     Running            0          8m27s
myminio-3                                  1/1     Running            0          8m27s

marcosbc · 2021-03-30T08:41:50Z

Also, make sure that there isn't any PVC from a previous deployment. If not, it may fail due to deploying with the wrong credentials, causing errors like:

API: SYSTEM()
Time: 08:39:27 UTC 03/30/2021
Error: Marking http://myminio-1.myminio-headless.default.svc.cluster.local:9000/minio/storage/data/v29 temporary offline; caused by Post "http://myminio-1.myminio-headless.default.svc.cluster.local:9000/minio/storage/data/v29/readall?disk-id=&file-path=format.json&volume=.minio.sys": lookup myminio-1.myminio-headless.default.svc.cluster.local on 10.30.240.10:53: no such host (*fmt.wrapError)
       6: cmd/rest/client.go:138:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:151:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:471:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:405:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:325:cmd.loadFormatErasureAll.func1()
       1: pkg/sync/errgroup/errgroup.go:122:errgroup.(*Group).Go.func1()

ethernoy · 2021-03-30T10:06:18Z

Hi @marcosbc

Attached is the content of value.yaml I use in the deployment:

global:
  imagePullSecrets: 
  - mySecret
  storageClass: myStorageClass
image:
  registry: myregistry
  repository: observability/bitnami/minio
  tag: 2021.3.26-debian-10-r1
  pullPolicy: Always
  debug: true
clientImage:
  registry: myregistry
  repository: observability/bitnami/minio-client
  tag: 2021.3.23-debian-10-r5
mode: distributed
accessKey:
  password: thanos123
  forcePassword: true
secretKey:
  password: thanos123
  forcePassword: true
defaultBuckets: "thanos"
statefulset:
  updateStrategy: RollingUpdate
  podManagementPolicy: Parallel
  replicaCount: 4
  zones: 1
  drivesPerNode: 1
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 0
resources:
  limits:
    cpu: 300m
    memory: 512Mi
  requests:
    cpu: 256m
    memory: 256Mi
persistence:
  size: 10Gi

here is the content of myStorageClass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2021-03-25T03:26:52Z"
  name: myStorageClass
parameters:
  svStorageClass: myStorageClass
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

marcosbc · 2021-03-30T14:21:35Z

Please also share more information like the deployment error you are seeing before the pod gets restarted.

Did you check if there could exist an existing PVC where Minio data is stored with different credentials than the ones you set?

ethernoy · 2021-03-31T02:48:31Z

Please also share more information like the deployment error you are seeing before the pod gets restarted.

Did you check if there could exist an existing PVC where Minio data is stored with different credentials than the ones you set?

Most of the MinIO ends with the following log pattern, while seldomly it has other logs before it terminates, but I did not manage to capture one yet.

API: SYSTEM()
Time: 02:43:48 UTC 03/31/2021
Error: Marking http://minio-test-2.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29 temporary offline; caused by Post "http://minio-test-2.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 192.168.12.40:9000: connect: connection refused (*fmt.wrapError)
       6: cmd/rest/client.go:138:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:151:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:471:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:405:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:325:cmd.loadFormatErasureAll.func1()
       1: pkg/sync/errgroup/errgroup.go:122:errgroup.(*Group).Go.func1()
Waiting for a minimum of 2 disks to come online (elapsed 8s)
 02:43:48.95 INFO  ==> Adding local Minio host to 'mc' configuration...
API: SYSTEM()
Time: 02:43:49 UTC 03/31/2021
Error: Marking http://minio-test-2.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29 temporary offline; caused by Post "http://minio-test-2.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 192.168.12.40:9000: connect: connection refused (*fmt.wrapError)
       6: cmd/rest/client.go:138:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:151:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:471:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:405:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:325:cmd.loadFormatErasureAll.func1()
       1: pkg/sync/errgroup/errgroup.go:122:errgroup.(*Group).Go.func1()
API: SYSTEM()
Time: 02:43:49 UTC 03/31/2021
Error: Marking http://minio-test-0.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29 temporary offline; caused by Post "http://minio-test-0.minio-test-headless.observability.svc.cluster.local:9000/minio/storage/data/v29/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 192.168.9.48:9000: connect: connection refused (*fmt.wrapError)
       6: cmd/rest/client.go:138:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:151:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:471:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:405:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:325:cmd.loadFormatErasureAll.func1()
       1: pkg/sync/errgroup/errgroup.go:122:errgroup.(*Group).Go.func1()
Waiting for a minimum of 2 disks to come online (elapsed 8s)
API: SYSTEM()
Time: 02:43:49 UTC 03/31/2021
Error: Access Denied. (*errors.errorString)
       requestHeaders={"method":"GET","reqURI":"/minio/admin/v3/info","header":{"Host":["localhost:9000"],"User-Agent":["MinIO (linux; amd64) madmin-go/0.0.1 mc/DEVELOPMENT.2021-03-23T09-13-19Z"],"X-Amz-Content-Sha256":["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]}}
       4: cmd/auth-handler.go:143:cmd.validateAdminSignature()
       3: cmd/auth-handler.go:159:cmd.checkAdminRequestAuth()
       2: cmd/admin-handlers.go:1520:cmd.adminAPIHandlers.ServerInfoHandler()
       1: net/http/server.go:2069:http.HandlerFunc.ServeHTTP()
 02:43:49.33 INFO  ==> MinIO is already stopped...
stream closed

marcosbc · 2021-03-31T16:47:38Z

Hi @ethernoy, the error looks like it could be related to there being an existing PVC:

Error: Access Denied. (*errors.errorString)

I still haven't got confirmation from your side that you've checked if that could be the case. You can get the list of PVCs with kubectl get pvc.

Could you recommend in another namespace and/or release name and check if it works? Make sure the release name is unique in the namespace (i.e. miniotest-123-unique) or you will get the same errors.

ethernoy · 2021-04-01T01:58:39Z

Hi @ethernoy, the error looks like it could be related to there being an existing PVC:
Error: Access Denied. (*errors.errorString)
I still haven't got confirmation from your side that you've checked if that could be the case. You can get the list of PVCs with kubectl get pvc.

Could you recommend in another namespace and/or release name and check if it works? Make sure the release name is unique in the namespace (i.e. miniotest-123-unique) or you will get the same errors.

I just tested two cases:

uninstall minio on the same namespace, delete all related pvc, then reinstall
install minio on a different namespace using a different release name

Both test resulted in the same “Access Denied” error we discussed above

marcosbc · 2021-04-01T10:39:05Z

Hi @ethernoy, I'm checking your configuration and I don't understand this:

securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 0

That will cause Minio to run as a root user. Currently it is not supported by the Docker image as there seems to be a bug where the minio user is not created before starting the container:

 10:35:36.00 INFO  ==> ** Starting MinIO **
error: failed switching to "minio": unable to find user minio: no matching entries in passwd file

If I remove that configuration, I'm able to workaround that error. Could you try it?

ethernoy · 2021-04-01T12:48:24Z

Hi @ethernoy, I'm checking your configuration and I don't understand this:
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 0
That will cause Minio to run as a root user. Currently it is not supported by the Docker image as there seems to be a bug where the minio user is not created before starting the container:
 10:35:36.00 INFO  ==> ** Starting MinIO **
error: failed switching to "minio": unable to find user minio: no matching entries in passwd file
If I remove that configuration, I'm able to workaround that error. Could you try it?

I believe it is not related to runAsRoot configuration. I tested under following cases:

Install as root, encounter access denied error, then restart service as non-root, encounter file unable to access error.
Install as non-root, encounter access denied error

Here is the non-root values.yaml I used:

global:
  imagePullSecrets: 
  - platform-tool-docker-repo
  storageClass: dev-cld-st01-storage-policy
image:
  registry: {repository_link}
  repository: observability/bitnami/minio
  tag: 2021.3.26-debian-10-r1
  pullPolicy: Always
  debug: true
clientImage:
  registry: {repository_link}
  repository: observability/bitnami/minio-client
  tag: 2021.3.23-debian-10-r5
mode: distributed
accessKey:
  password: thanos123
  forcePassword: true
secretKey:
  password: thanos123
  forcePassword: true
defaultBuckets: "thanos"
statefulset:
  updateStrategy: RollingUpdate
  podManagementPolicy: Parallel
  replicaCount: 4
  zones: 1
  drivesPerNode: 1
resources:
  limits:
    cpu: 300m
    memory: 512Mi
  requests:
    cpu: 256m
    memory: 256Mi
persistence:
  size: 10Gi

Installing Bitnami MinIO using this values.yaml still results in access denied error:

marcosbc · 2021-04-02T10:05:36Z

Hi, I'm going to forward this case to @juan131 who has more experience with MinIO. I'm finding some issues myself, although not related to your error (which I'm able to get past without issues).

In the meantime, it would be great if you could share more specs on your Kubernetes cluster. For instance, is it a vanilla Kubernetes cluster or are you running on a Kubernetes distribution? Could it also be that you are running a MinIO image based on Photon, from TAC?

juan131 · 2021-04-05T07:26:02Z

Hi @ethernoy

I agree with @marcosbc that the "securityContext" shouldn't be forcing the container to run as user "0", so please ensure you're including the section below in your values.yaml:

securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001

You'll have to ensure that your cluster has support for changing the ownership and permissions for the contents of each volume. See:

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods

Also, we're finding some issues with the "default buckets" feature in distributed mode. Therefore, I recommend you to remove the parameter defaultBuckets from your parameters and create manually your buckets after MinIO cluster is up and running.

ethernoy · 2021-04-07T02:46:26Z

Hi @ethernoy

I agree with @marcosbc that the "securityContext" shouldn't be forcing the container to run as user "0", so please ensure you're including the section below in your values.yaml:
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
You'll have to ensure that your cluster has support for changing the ownership and permissions for the contents of each volume. See:

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods

Also, we're finding some issues with the "default buckets" feature in distributed mode. Therefore, I recommend you to remove the parameter defaultBuckets from your parameters and create manually your buckets after MinIO cluster is up and running.

Hi, just double confirmed, when deploying with the default securityContext and disabled default bucket, Bitnami MinIO can work normally.

juan131 · 2021-04-07T07:09:31Z

Great!! I'll set a reminder to myself to see how can we modify the approach we're using to create the "default buckets" with the distributed mode. It's clearly not working as expected.

ethernoy · 2021-04-08T02:16:26Z

Great!! I'll set a reminder to myself to see how can we modify the approach we're using to create the "default buckets" with the distributed mode. It's clearly not working as expected.

I am curious, is this issue related only to the Bitnami MinIO chart and can be solved by chart modification solely, or is it related to the Bitnami MinIO docker image too?

juan131 · 2021-04-08T07:19:26Z

I'd say that both @ethernoy

It can be solved by improving the logic in the Bitnami MinIO container image which is not properly managing the buckets' creation when a distributed mode is used.
However, we can also take a completely different approach and delegate the default buckets creation on a Kubernetes job that creates them once the MinIO cluster is up and ready. In this case, we will implement the solution in the Minio chart without modifying the current logic of the container.

github-actions · 2021-04-24T01:24:25Z

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

ZILosoft · 2022-04-21T16:27:56Z

any news?

carrodher · 2022-04-22T06:51:35Z

Unfortunately, there was no internal progress on this task and I'm afraid that, if it was not prioritized during this time, there's not much chance we'll be working on it in the short term. Since we are a small team maintaining a lot of assets it is difficult to find the bandwidth to implement all the requests.

Being said that, thanks for reporting this issue and to be on top of it. Would you like to contribute by creating a PR to solve the issue? The Bitnami team will be happy to review it and provide feedback. Here you can find the contributing guidelines.

github-actions · 2022-05-08T01:31:44Z

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions · 2022-05-13T01:32:43Z

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

papierkorp · 2023-04-14T08:50:26Z

I had this problem with a minikube, since i just needed it for testing purposes I added --set persistence.enabled="false"

The Full Command being:

helm install minio bitnami/minio --namespace minio --create-namespace --set image.debug="true" --set service.type="ClusterIP" --set persistence.enabled="false"

gruberdev · 2023-11-09T15:35:19Z

I had this problem with a minikube, since i just needed it for testing purposes I added --set persistence.enabled="false"

The Full Command being:
helm install minio bitnami/minio --namespace minio --create-namespace --set image.debug="true" --set service.type="ClusterIP" --set persistence.enabled="false"

If there's no storage system or persistence, it makes sense that this error would not occur.

I would argue it is not relevant to the issue discussed here.

github-actions bot added the stale 15 days without activity label Apr 24, 2021

juan131 added on-hold Issues or Pull Requests with this label will never be considered stale and removed stale 15 days without activity labels Apr 26, 2021

sagor999 mentioned this issue Jan 19, 2022

[installer] minio pod on k3s in crashloop gitpod-io/gitpod#7123

Closed

carrodher removed the on-hold Issues or Pull Requests with this label will never be considered stale label Apr 22, 2022

github-actions bot added the stale 15 days without activity label May 8, 2022

github-actions bot closed this as completed May 13, 2022

github-actions bot added the triage Triage is needed label Nov 9, 2023

bitnami-bot assigned carrodher Nov 9, 2023

carrodher added the minio label Nov 9, 2023

github-actions bot added the solved label Nov 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bitnami/minio] Access Denied when Deploying Bitnami/MinIO 2021.3.26-debian-10-r1 #5951

[bitnami/minio] Access Denied when Deploying Bitnami/MinIO 2021.3.26-debian-10-r1 #5951

ethernoy commented Mar 30, 2021

marcosbc commented Mar 30, 2021

marcosbc commented Mar 30, 2021

ethernoy commented Mar 30, 2021

marcosbc commented Mar 30, 2021

ethernoy commented Mar 31, 2021

marcosbc commented Mar 31, 2021

ethernoy commented Apr 1, 2021

marcosbc commented Apr 1, 2021

ethernoy commented Apr 1, 2021

marcosbc commented Apr 2, 2021

juan131 commented Apr 5, 2021

ethernoy commented Apr 7, 2021

juan131 commented Apr 7, 2021

ethernoy commented Apr 8, 2021

juan131 commented Apr 8, 2021

github-actions bot commented Apr 24, 2021

ZILosoft commented Apr 21, 2022

carrodher commented Apr 22, 2022

github-actions bot commented May 8, 2022

github-actions bot commented May 13, 2022

papierkorp commented Apr 14, 2023

gruberdev commented Nov 9, 2023

[bitnami/minio] Access Denied when Deploying Bitnami/MinIO 2021.3.26-debian-10-r1 #5951

[bitnami/minio] Access Denied when Deploying Bitnami/MinIO 2021.3.26-debian-10-r1 #5951

Comments

ethernoy commented Mar 30, 2021

marcosbc commented Mar 30, 2021

marcosbc commented Mar 30, 2021

ethernoy commented Mar 30, 2021

marcosbc commented Mar 30, 2021

ethernoy commented Mar 31, 2021

marcosbc commented Mar 31, 2021

ethernoy commented Apr 1, 2021

marcosbc commented Apr 1, 2021

ethernoy commented Apr 1, 2021

marcosbc commented Apr 2, 2021

juan131 commented Apr 5, 2021

ethernoy commented Apr 7, 2021

juan131 commented Apr 7, 2021

ethernoy commented Apr 8, 2021

juan131 commented Apr 8, 2021

github-actions bot commented Apr 24, 2021

ZILosoft commented Apr 21, 2022

carrodher commented Apr 22, 2022

github-actions bot commented May 8, 2022

github-actions bot commented May 13, 2022

papierkorp commented Apr 14, 2023

gruberdev commented Nov 9, 2023