S3 blocks storage minimum permissions

[jgutschon]

I am running Grafana Mimir in Kubernetes using the Helm chart (v2.1.0) and I'm using S3 for the TSDB blocks storage, however, I am now having issues getting some services to connect to the S3 bucket. All other services are connecting successfully, but the store-gateway is giving "Access Denied" errors when attempting a connection to the bucket.

The service account role has the following IAM permissions associated with it:

"s3:ListBucket",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"

I'm wondering what the minimum required IAM permissions are for using S3 as the TSDB blocks storage since there doesn't seem to be any documentation on this.

[pracucci]

Hi @jgutschon!

On top on my mind, I think you need to also add GetObjectAttributes.

the store-gateway is giving "Access Denied" errors when attempting a connection to the bucket

Could you share the entire log line, please? I would like to check which actual operation the store-gateway was trying to do.

[jgutschon]

Awesome thanks @pracucci, that seems to have fixed it.

Here are the pod logs before adding the extra permission:

level=info ts=2022-06-29T15:09:48.518650354Z caller=main.go:201 msg="Starting application" version="(version=2.1.0, branch=HEAD, revision=840c686)"
level=info ts=2022-06-29T15:09:48.521829554Z caller=server.go:285 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-06-29T15:09:48.54259356Z caller=memberlist_client.go:395 msg="Using memberlist cluster node name" name=mimir-store-gateway-0-1f2edd19
level=info ts=2022-06-29T15:09:48.54885223Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-06-29T15:09:48.557942463Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-06-29T15:09:48.565971036Z caller=memcached.go:60 msg="created memcached index cache"
level=info ts=2022-06-29T15:09:48.568891282Z caller=module_service.go:64 msg=initialising module=activity-tracker
level=info ts=2022-06-29T15:09:48.568990004Z caller=module_service.go:64 msg=initialising module=sanity-check
level=info ts=2022-06-29T15:09:48.569033274Z caller=sanity_check.go:33 msg="Checking directories read/write access"
level=info ts=2022-06-29T15:09:48.569201458Z caller=sanity_check.go:38 msg="Directories read/write access successfully checked"
level=info ts=2022-06-29T15:09:48.569252139Z caller=sanity_check.go:40 msg="Checking object storage config"
level=info ts=2022-06-29T15:09:48.573441398Z caller=memberlist_client.go:514 msg="joined memberlist cluster" reached_nodes=10
level=warn ts=2022-06-29T15:09:48.646624515Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:09:49.811925053Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:09:52.142210152Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:09:56.843194285Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:01.313539557Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:06.244362047Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:11.307130401Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:15.464319286Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:20.029295101Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:24.710811447Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:29.538219655Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:33.722109524Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:38.088995196Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:42.668853167Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:47.195281329Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:51.838359322Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:10:56.391547176Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:11:00.945584054Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:11:05.451054534Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=warn ts=2022-06-29T15:11:09.600159434Z caller=sanity_check.go:118 msg="Unable to successfully connect to configured object storage" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=error ts=2022-06-29T15:11:09.600235056Z caller=sanity_check.go:42 msg="Unable to successfully connect to configured object storage" err="blocks storage: unable to successfully send a request to object storage: Access Denied"
level=error ts=2022-06-29T15:11:09.600289367Z caller=mimir.go:472 msg="module failed" module=sanity-check err="invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: Access Denied"
level=error ts=2022-06-29T15:11:09.600357308Z caller=mimir.go:472 msg="module failed" module=server err="failed to start server, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: Access Denied"
level=error ts=2022-06-29T15:11:09.600393849Z caller=mimir.go:472 msg="module failed" module=runtime-config err="failed to start runtime-config, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: Access Denied"
level=error ts=2022-06-29T15:11:09.600420299Z caller=mimir.go:472 msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module server, which has failed: context canceled"
level=error ts=2022-06-29T15:11:09.60045097Z caller=mimir.go:472 msg="module failed" module=memberlist-kv err="failed to start memberlist-kv, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: blocks storage: unable to successfully send a request to object storage: Access Denied"
level=info ts=2022-06-29T15:11:09.603702722Z caller=module_service.go:96 msg="module stopped" module=activity-tracker
level=error ts=2022-06-29T15:11:09.603897936Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:507\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:203\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

After adding the permission, all pods appear to be healthy and there aren't any more Access Denied errors, however, I am now getting warnings from almost all of the pods about the memberlist. Here is one from the store-gateway:

level=warn ts=2022-06-29T15:24:00.245984133Z caller=tcp_transport.go:428 component="memberlist TCPTransport" msg="WriteTo failed" addr=172.29.18.146:7946 err="dial tcp 172.29.18.146:7946: i/o timeout"