Mimir and S3 endpoint #4215

ioricloud · 2023-02-09T15:39:04Z

ioricloud
Feb 9, 2023

Hi people.

I am trying to deploy mimir in my observability infrastructure and I have big problem here.

In config.yml I put endpoint s3.us-east-1.amazonaws.com

but always got s3 endpoint via ecs container.

ts=2023-02-09T15:30:50.964964464Z caller=log.go:65 level=error msg="error running application" err="no s3 endpoint in config file\ngithub.com/thanos-io/objstore/providers/s3.validate\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:347\ngithub.com/thanos-io/objstore/providers/s3.NewBucketWithConfig\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:218\ngithub.com/grafana/mimir/pkg/storage/bucket/s3.NewBucketClient\n\t/__w/mimir/mimir/pkg/storage/bucket/s3/bucket_client.go:22\ngithub.com/grafana/mimir/pkg/storage/bucket.NewClient\n\t/__w/mimir/mimir/pkg/storage/bucket/client.go:163\ngithub.com/grafana/mimir/pkg/ruler.NewRuleStore\n\t/__w/mimir/mimir/pkg/ruler/storage.go:32\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initRulerStorage\n\t/__w/mimir/mimir/pkg/mimir/modules.go:597\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:752\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:752\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

What do I need to do right now?

Mimir image 2.6.0

56quarters · 2023-02-09T15:42:47Z

56quarters
Feb 9, 2023
Maintainer

Can you post your configuration (with secrets removed)? The endpoint you gave is correct so it seems like there's something else going on here.

0 replies

ioricloud · 2023-02-09T16:18:23Z

ioricloud
Feb 9, 2023
Author

I found a problem, mimir image 2.6.0 has a big problem, I am using the 2.5.0 and is up right now. :( Em qui., 9 de fev. de 2023 às 12:42, Nick Pillitteri < ***@***.***> escreveu:

…

Can you post your configuration (with secrets removed)? The endpoint you gave is correct so it seems like there's something else going on here. — Reply to this email directly, view it on GitHub <#4215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEX6XI76PYCUNXW3S32TDWWUGADANCNFSM6AAAAAAUWWYFSI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Henrique Oliveira DevOps Engineer/SRE. Network Computer Graduated - FIC Estácio. DevOps/SRE PosDegree - IGTI Education. Site: elementar.me Blog: ioricloud.gitlab.io/hugo-blog/

0 replies

pracucci · 2023-02-09T17:47:39Z

pracucci
Feb 9, 2023
Maintainer

mimir image 2.6.0 has a big problem

What's going on with Mimir 2.6.0? Can you share more details, please?

0 replies

ioricloud · 2023-02-09T20:45:34Z

ioricloud
Feb 9, 2023
Author

I will get log errors and put here. when I use previous version, ok, mimir is up, in 2.6.0 always stay down. Em qui., 9 de fev. de 2023 às 14:47, Marco Pracucci < ***@***.***> escreveu:

…

mimir image 2.6.0 has a big problem What's going on with Mimir 2.6.0? Can you share more details, please? — Reply to this email directly, view it on GitHub <#4215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEX6TGWS4N5V5RUERT3W3WWUUUNANCNFSM6AAAAAAUWWYFSI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Henrique Oliveira DevOps Engineer/SRE. Network Computer Graduated - FIC Estácio. DevOps/SRE PosDegree - IGTI Education. Site: elementar.me Blog: ioricloud.gitlab.io/hugo-blog/

0 replies

ioricloud · 2023-02-09T20:49:32Z

ioricloud
Feb 9, 2023
Author

docker run -d --name mimir -p 9009:9009 -v ./mimir.yml:/etc/mimir.yml grafana/mimir:2.5.0 -config.file=/etc/mimir.yml && docker logs -f mimir && docker rm mimir
e1071eae1046fbc6f7e0093f6fb93b0d4e6745ec5ca2effd7cd2bbff50f9c06a
level=info ts=2023-02-09T20:30:54.074877896Z caller=main.go:210 msg="Starting application" version="(version=2.5.0, branch=HEAD, revision=25533fd)"
level=info ts=2023-02-09T20:30:54.076226843Z caller=server.go:323 http=[::]:9009 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2023-02-09T20:30:54.084118834Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=gm-mimir-qa-260dba09
level=info ts=2023-02-09T20:30:54.086186496Z caller=ingester.go:337 msg="TSDB idle compaction timeout set" timeout=1h11m43.251337807s
level=info ts=2023-02-09T20:30:54.087470848Z caller=inmemory.go:173 msg="created in-memory index cache" maxItemSizeBytes=134217728 maxSizeBytes=1073741824 maxItems=maxInt
level=info ts=2023-02-09T20:30:54.089346105Z caller=mapper.go:52 msg="cleaning up mapped rules directory" path=./data-ruler/
level=info ts=2023-02-09T20:30:54.090818279Z caller=modules.go:416 msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
level=info ts=2023-02-09T20:30:54.090847174Z caller=worker.go:130 msg="Starting querier worker connected to query-frontend" frontend=127.0.0.1:9095
level=warn ts=2023-02-09T20:30:54.091160391Z caller=multitenant.go:151 msg="The configured Alertmanager HTTP prefix '/alertmanager' is different than the path specified in the external URL 'http://alertmanager-qa.greenmile.com': the Alertmanager UI and API may not work as expected unless you have a reverse proxy exposing the Alertmanager endpoints under '/alertmanager' prefix"
level=info ts=2023-02-09T20:30:54.093135965Z caller=module_service.go:82 msg=initialising module=activity-tracker
level=info ts=2023-02-09T20:30:54.093158467Z caller=module_service.go:82 msg=initialising module=sanity-check
level=info ts=2023-02-09T20:30:54.093203552Z caller=sanity_check.go:32 msg="Checking directories read/write access"
level=info ts=2023-02-09T20:30:54.093692263Z caller=sanity_check.go:37 msg="Directories read/write access successfully checked"
level=info ts=2023-02-09T20:30:54.093707033Z caller=sanity_check.go:39 msg="Checking object storage config"
level=warn ts=2023-02-09T20:31:24.095491285Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:31:55.43313336Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:32:29.370588978Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:33:04.316950196Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:33:38.733292455Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:34:13.620046015Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:34:48.346339906Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:35:23.288365554Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:35:58.098742307Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:36:32.303218593Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:37:07.040536089Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:37:41.247478856Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:38:15.836728431Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:38:50.366901654Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:39:24.47924335Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:39:58.516868027Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:40:32.526610603Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:41:06.610618716Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:41:40.701716034Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=warn ts=2023-02-09T20:42:15.415469089Z caller=sanity_check.go:117 msg="Unable to successfully connect to configured object storage" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=error ts=2023-02-09T20:42:15.415523392Z caller=sanity_check.go:41 msg="Unable to successfully connect to configured object storage" err="3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=error ts=2023-02-09T20:42:15.415614478Z caller=mimir.go:791 msg="module failed" module=sanity-check err="invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=error ts=2023-02-09T20:42:15.415656143Z caller=mimir.go:791 msg="module failed" module=memberlist-kv err="failed to start memberlist-kv, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=error ts=2023-02-09T20:42:15.41568164Z caller=mimir.go:791 msg="module failed" module=querier err="failed to start querier, because it depends on module store-queryable, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415694559Z caller=mimir.go:791 msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415704672Z caller=mimir.go:791 msg="module failed" module=compactor err="failed to start compactor, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415718105Z caller=mimir.go:791 msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415731802Z caller=mimir.go:791 msg="module failed" module=ring err="failed to start ring, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.41574375Z caller=mimir.go:791 msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.41575494Z caller=mimir.go:791 msg="module failed" module=query-frontend err="failed to start query-frontend, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415767681Z caller=mimir.go:791 msg="module failed" module=ingester-service err="failed to start ingester-service, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415778848Z caller=mimir.go:791 msg="module failed" module=server err="failed to start server, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: Get \"http://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; alertmanager storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get \"https://gm-mimir-qa.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded"
level=error ts=2023-02-09T20:42:15.415792193Z caller=mimir.go:791 msg="module failed" module=alertmanager err="failed to start alertmanager, because it depends on module memberlist-kv, which has failed: context canceled"
level=error ts=2023-02-09T20:42:15.415806987Z caller=mimir.go:791 msg="module failed" module=ruler err="failed to start ruler, because it depends on module server, which has failed: context canceled"
level=info ts=2023-02-09T20:42:15.417808001Z caller=module_service.go:114 msg="module stopped" module=activity-tracker
level=info ts=2023-02-09T20:42:15.417845713Z caller=mimir.go:780 msg="Application stopped"
level=error ts=2023-02-09T20:42:15.417898391Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:834\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

0 replies

56quarters · 2023-02-09T20:57:31Z

56quarters
Feb 9, 2023
Maintainer

Can you please share your mimir.yml configuration file?

0 replies

ioricloud · 2023-02-09T22:12:10Z

ioricloud
Feb 9, 2023
Author

yes
https://pastebin.com/bCQ7dV08

0 replies

pracucci · 2023-02-15T08:37:15Z

pracucci
Feb 15, 2023
Maintainer

Could you check if you're hitting this issue, please? The discussion also shows how to fix it:
#2613 (comment)

0 replies

ioricloud · 2023-02-15T12:05:42Z

ioricloud
Feb 15, 2023
Author

Hi Marco, I didn't understand this thread, sorry. Let me explain my strategy. I am using Mimir on Amazon ECS, about this, I have a role with S3 admin via policy, I attached it via task definition when I deploy my infrastructure. In this case, I don't need to create credentials for S3. What must I do? Best Regards. Em qua., 15 de fev. de 2023 às 05:37, Marco Pracucci < ***@***.***> escreveu:

…

Could you check if you're hitting this issue, please? The discussion also shows how to fix it: #2613 (comment) <#2613 (comment)> — Reply to this email directly, view it on GitHub <#4215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEX6R4C64BW5SUAHM23MDWXSIUNANCNFSM6AAAAAAUWWYFSI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Henrique Oliveira DevOps Engineer/SRE. Network Computer Graduated - FIC Estácio. DevOps/SRE PosDegree - IGTI Education. Site: elementar.me Blog: ioricloud.gitlab.io/hugo-blog/

2 replies

ioricloud Feb 15, 2023
Author

In my config I dont use dualstack too.

pracucci Feb 24, 2023
Maintainer

As previously mentioned, I suggest to check if you're hit by this. The linked comment is almost self explanatory: there's a link to the AWS doc explaining the issue and a command to change the behavior.

ioricloud · 2023-03-01T22:10:10Z

ioricloud
Mar 1, 2023
Author

I put in mimir.yaml region again but I got some warn. level=warn ts=2023-03-01T22:06:39.676567576Z caller=sanity_check.go:115 msg="Unable to successfully connect to configured object storage (will retry)" err="3 errors: blocks storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; alertmanager storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; ruler storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'" but is not my error level=error ts=2023-03-01T22:08:01.499631575Z caller=mimir.go:791 msg="module failed" module=querier err="failed to start querier, because it depends on module server, which has failed: invalid service state: Failed, expected: Running, failure: failed to start server, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; alertmanager storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; ruler storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'" level=error ts=2023-03-01T22:08:01.499771587Z caller=mimir.go:791 msg="module failed" module=alertmanager err="failed to start alertmanager, because it depends on module server, which has failed: invalid service state: Failed, expected: Running, failure: failed to start server, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; alertmanager storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; ruler storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'" level=error ts=2023-03-01T22:08:01.500094736Z caller=mimir.go:791 msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; alertmanager storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; ruler storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'" level=error ts=2023-03-01T22:08:01.50045999Z caller=mimir.go:791 msg="module failed" module=ruler err="failed to start ruler, because it depends on module store-queryable, which has failed: invalid service state: Failed, expected: Running, failure: failed to start store-queryable, because it depends on module sanity-check, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: 3 errors: blocks storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; alertmanager storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'; ruler storage: unable to successfully send a request to object storage: The authorization header is malformed; the region 'us-east' is wrong; expecting 'us-east-1'" Em sex., 24 de fev. de 2023 às 14:19, Marco Pracucci < ***@***.***> escreveu:

…

As previously mentioned, I suggest to check if you're hit by this <#2613 (comment)>. The linked comment is almost self explanatory: there's a link to the AWS doc explaining the issue and a command to change the behavior. — Reply to this email directly, view it on GitHub <#4215 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEX6UWMZUCNE74MFBTHFDWZDUSTANCNFSM6AAAAAAUWWYFSI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Henrique Oliveira DevOps Engineer/SRE. Network Computer Graduated - FIC Estácio. DevOps/SRE PosDegree - IGTI Education. Site: elementar.me Blog: ioricloud.gitlab.io/hugo-blog/

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mimir and S3 endpoint #4215

{{title}}

Replies: 10 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Mimir and S3 endpoint #4215

ioricloud Feb 9, 2023

Replies: 10 comments · 2 replies

56quarters Feb 9, 2023 Maintainer

ioricloud Feb 9, 2023 Author

pracucci Feb 9, 2023 Maintainer

ioricloud Feb 9, 2023 Author

ioricloud Feb 9, 2023 Author

56quarters Feb 9, 2023 Maintainer

ioricloud Feb 9, 2023 Author

pracucci Feb 15, 2023 Maintainer

ioricloud Feb 15, 2023 Author

ioricloud Feb 15, 2023 Author

pracucci Feb 24, 2023 Maintainer

ioricloud Mar 1, 2023 Author

ioricloud
Feb 9, 2023

Replies: 10 comments 2 replies

56quarters
Feb 9, 2023
Maintainer

ioricloud
Feb 9, 2023
Author

pracucci
Feb 9, 2023
Maintainer

ioricloud
Feb 9, 2023
Author

ioricloud
Feb 9, 2023
Author

56quarters
Feb 9, 2023
Maintainer

ioricloud
Feb 9, 2023
Author

pracucci
Feb 15, 2023
Maintainer

ioricloud
Feb 15, 2023
Author

ioricloud Feb 15, 2023
Author

pracucci Feb 24, 2023
Maintainer

ioricloud
Mar 1, 2023
Author