S3: Unable to sanity check object store #2613
-
Describe the bugWhen starting mimir with S3 configured as the block store, the sanity check is failing on our environment with a context deadline exeeded error message. level=info ts=2022-07-30T15:29:52.08092532Z caller=sanity_check.go:40 msg="Checking object storage config" Mimir is running inside docker using docker-compose, on an IPv4 AWS VPC. To ReproduceSteps to reproduce the behavior: Uncertain Expected behaviorThe same configuration applied to Tempo is working correctly. I would expect the same to work here. Environment
Additional ContextThe S3 endpoint is configured to use the non-dualstack endpoint, but the error message above seems to be wanting the dualstack endpoint. I'm wondering if its trying to use IPv6, which isn't supported in our environment.
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
Okay, I figured out that this is an issue with fetching the credentials. After putting in a lot of logging statements, I found that the call to fetchIMDSToken in https://github.com/grafana/mimir/blob/main/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go#L292 is hanging. When this call times out, after the default of 30 seconds, it will return an empty string as an timeout error will occur. I think the sanity check cancels the lookup earlier than the 30 seconds, so it doesn't get the chance to proceed. I tried increasing this up to 60, but didn't seem to make any difference. https://github.com/grafana/mimir/blob/main/pkg/mimir/sanity_check.go#L150 I couldn't see any in tempo does the same timeout check, so I expect that is why it works. |
Beta Was this translation helpful? Give feedback.
-
I found out that Amazon monitor the hop count when connecting to the metadata service for a PUT request. The default is 1, whereas I need 2 hops since I'm running inside docker. This can be modified using the CLI to increase the hop limit.
|
Beta Was this translation helpful? Give feedback.
I found out that Amazon monitor the hop count when connecting to the metadata service for a PUT request. The default is 1, whereas I need 2 hops since I'm running inside docker. This can be modified using the CLI to increase the hop limit.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html#imds-considerations