[main branch] Query - GoPanic after fix for NoStore API match

[ahurtaud]

Thanos, Prometheus and Golang version used:
main-2023-02-04-48e82c5

What happened:
Hello, I tried to load version main-2023-02-04-48e82c5 which contains the fix for #5862 as we were stuck in v0.28.1. We are using a tree of queriers so we were impacted by this issue.
The issue is indeed fixed, but the querier CPU went crazy and this go panic happened multiple times:

go panic

2023/02/06 09:10:35 http: panic serving 10.240.20.133:44096: duplicate metrics collector registration attempted
goroutine 217972981 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x2288980, 0xc05ee64000})
	/usr/local/go/src/runtime/panic.go:890 +0x262
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc184709e00, {0xc0abe99d60?, 0x1, 0x0?})
	/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/wrap.go:106 +0x151
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGauge({{0x2a8b3f0?, 0xc184709e00?}}, {{0x0, 0x0}, {0x0, 0x0}, {0x254ea87, 0x10}, {0x258bd2c, 0x25}, ...})
	/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promauto/auto.go:297 +0xfd
github.com/thanos-io/thanos/pkg/gate.New({0x2a8b3f0, 0xc184709e00}, 0x14)
	/app/pkg/gate/gate.go:90 +0x7e
github.com/thanos-io/thanos/pkg/query.NewQueryableCreator.func1.1()
	/app/pkg/query/querier.go:91 +0x28
github.com/thanos-io/thanos/pkg/query.(*queryable).Querier(0xc032914b40, {0x2a91c48, 0xc07bb9f7a0}, 0x2441180?, 0xc0abe99d00?)
	/app/pkg/query/querier.go:121 +0x58
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc000801b00, {0x2a91c48, 0xc07bb9f710}, 0xc24d8f8cb0, 0xc2146fdf40)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.41.0/promql/engine.go:625 +0x158
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc000801b00, {0x2a91c48, 0xc07bb9f710}, 0xc24d8f8cb0)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.41.0/promql/engine.go:605 +0x4ec
github.com/prometheus/prometheus/promql.(*query).Exec(0xc24d8f8cb0, {0x2a91c48, 0xc184709dd0})
	/go/pkg/mod/github.com/prometheus/prometheus@v0.41.0/promql/engine.go:202 +0x1f5
github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).queryRange(0xc0006e7c20, 0xc21434d000)
	/app/pkg/api/query/v1.go:593 +0xf5b
github.com/thanos-io/thanos/pkg/api.GetInstr.func1.1({0x2a89500, 0xc213099c70}, 0x5?)
	/app/pkg/api/api.go:211 +0x50
net/http.HandlerFunc.ServeHTTP(0xc0604f2060?, {0x2a89500?, 0xc213099c70?}, 0x2a6e2e8?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1({0x2a89500?, 0xc213099c70}, 0xc21434d000)
	/app/pkg/logging/http.go:69 +0x3b8
net/http.HandlerFunc.ServeHTTP(0x2a91c48?, {0x2a89500?, 0xc213099c70?}, 0x2a6eb20?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1({0x2a89500, 0xc213099c70}, 0xc21434cf00)
	/app/pkg/server/http/middleware/request_id.go:40 +0x542
net/http.HandlerFunc.ServeHTTP(0x2058640?, {0x2a89500?, 0xc213099c70?}, 0x4?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x2a8cf80, 0xc03003e200}, 0x48fe01?)
	/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x26f
net/http.HandlerFunc.ServeHTTP(0x6f635f737463656c?, {0x2a8cf80?, 0xc03003e200?}, 0x727275636e6f635f?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/thanos-io/thanos/pkg/extprom/http.httpInstrumentationHandler.func1({0x7f6de3130158?, 0xc2146fde50}, 0xc21434cf00)
	/app/pkg/extprom/http/instrument_server.go:75 +0x10b
net/http.HandlerFunc.ServeHTTP(0x7f6de3130158?, {0x7f6de3130158?, 0xc2146fde50?}, 0xc184709b90?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7f6de3130158?, 0xc2146fde00?}, 0xc21434cf00)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:288 +0xc5
net/http.HandlerFunc.ServeHTTP(0x7f6de3130158?, {0x7f6de3130158?, 0xc2146fde00?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x7f6de3130158?, 0xc2146fddb0?}, 0xc21434cf00)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xb8
net/http.HandlerFunc.ServeHTTP(0x21937a0?, {0x7f6de3130158?, 0xc2146fddb0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/thanos-io/thanos/pkg/extprom/http.instrumentHandlerInFlight.func1({0x7f6de3130158, 0xc2146fddb0}, 0xc21434cf00)
	/app/pkg/extprom/http/instrument_server.go:162 +0x169
net/http.HandlerFunc.ServeHTTP(0x2a8ee40?, {0x7f6de3130158?, 0xc2146fddb0?}, 0xc075365698?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1({0x2a8ee40?, 0xc0201a28c0?}, 0xc21434cf00)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:238 +0xc5
net/http.HandlerFunc.ServeHTTP(0x2a91c48?, {0x2a8ee40?, 0xc0201a28c0?}, 0x3f43b20?)
	/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1({0x2a8ee40, 0xc0201a28c0}, 0xc21434cc00)
	/app/pkg/tracing/http.go:62 +0x9a2
github.com/prometheus/common/route.(*Router).handle.func1({0x2a8ee40, 0xc0201a28c0}, 0xc1c405cf00, {0x0, 0x0, 0x478aee?})
	/go/pkg/mod/github.com/prometheus/common@v0.39.1-0.20230110141620-846591a16635/route/route.go:83 +0x18d
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001b1800, {0x2a8ee40, 0xc0201a28c0}, 0xc1c405cf00)
	/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x81c
github.com/prometheus/common/route.(*Router).ServeHTTP(0xc0201a28c0?, {0x2a8ee40?, 0xc0201a28c0?}, 0x25491ef?)
	/go/pkg/mod/github.com/prometheus/common@v0.39.1-0.20230110141620-846591a16635/route/route.go:126 +0x26
net/http.(*ServeMux).ServeHTTP(0x0?, {0x2a8ee40, 0xc0201a28c0}, 0xc1c405cf00)
	/usr/local/go/src/net/http/server.go:2487 +0x149
net/http.serverHandler.ServeHTTP({0x2a877e0?}, {0x2a8ee40, 0xc0201a28c0}, 0xc1c405cf00)
	/usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc000b61900, {0x2a91c48, 0xc000236510})
	/usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3102 +0x4db

[yeya24]

Seems introduced in #6079. @GiedriusS Would you like to help fix this?

[GiedriusS]

Could you see if this is still reproducible after #6110? @ahurtaud

[ahurtaud]

Could you see if this is still reproducible after #6110? @ahurtaud

Sure I will try this afternoon and update here.

[ahurtaud]

Fixed by #6110 !!
Thank you very much @GiedriusS !!
Confirmed upgraded connected queriers from v0.28.1 to main-2023-02-08-83d2c14 is working perfectly !!

I let @yeya24 and @SuperQ decide if we want to release v0.30.3 with it or wait v0.31 release.

[yeya24]

This regression introduced in #6079 is on the main branch only so I think we can just do v0.31.