Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic loading store view #4914

Closed
gmintoco opened this issue Nov 30, 2021 · 4 comments
Closed

Panic loading store view #4914

gmintoco opened this issue Nov 30, 2021 · 4 comments
Labels
bug needs-more-info

Comments

@gmintoco
Copy link
Contributor

gmintoco commented Nov 30, 2021 •
edited
Loading

Thanos, Prometheus and Golang version used:
Thanos v0.23.1 docker image

Object Storage Provider:

GCS

What happened:

Panic when trying to load /stores. Web UI shows
image

Returning a 503 from /api/v1/stores

What you expected to happen:

Returns list of stores and their status

How to reproduce it (as minimally and precisely as possible):

I have a lot of store nodes (16) some of which aren't currently available (they come in and out of availability) I suspect this might have something to do with the issue but have attached the logs below as well.

Full logs to relevant components:

Logs 
    /home/circleci/project/pkg/extprom/http/instrument_server.go:108 +0x10f
    net/http.HandlerFunc.ServeHTTP(0xc0003eba10, 0x7f291c49c050, 0xc000a3d1d0, 0xc000db4a00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x7f291c49c050, 0xc000a3d180, 0xc000db4a00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:198 +0xee
    net/http.HandlerFunc.ServeHTTP(0xc0003ebd10, 0x7f291c49c050, 0xc000a3d180, 0xc000db4a00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x7f291c49c050, 0xc000a3d130, 0xc000db4a00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101 +0xdf
    net/http.HandlerFunc.ServeHTTP(0xc0003ebf80, 0x7f291c49c050, 0xc000a3d130, 0xc000db4a00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1(0x214e980, 0xc0008fa8c0, 0xc000db4a00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:165 +0xee
    net/http.HandlerFunc.ServeHTTP(0xc00048c2a0, 0x214e980, 0xc0008fa8c0, 0xc000db4a00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1(0x214e980, 0xc0008fa8c0, 0xc000db4900)
        /home/circleci/project/pkg/tracing/http.go:46 +0x54c
    github.com/prometheus/common/route.(*Router).handle.func1(0x214e980, 0xc0008fa8c0, 0xc000db4800, 0x0, 0x0, 0x0)
        /home/circleci/go/pkg/mod/github.com/prometheus/common@v0.30.0/route/route.go:83 +0x27f
    github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00098de60, 0x214e980, 0xc0008fa8c0, 0xc000db4800)
        /home/circleci/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0xc7e
    github.com/prometheus/common/route.(*Router).ServeHTTP(0xc000214b20, 0x214e980, 0xc0008fa8c0, 0xc000db4800)
        /home/circleci/go/pkg/mod/github.com/prometheus/common@v0.30.0/route/route.go:121 +0x4c
    net/http.(*ServeMux).ServeHTTP(0xc0005fadc0, 0x214e980, 0xc0008fa8c0, 0xc000db4800)
        /usr/local/go/src/net/http/server.go:2428 +0x1ad
    net/http.serverHandler.ServeHTTP(0xc00046c540, 0x214e980, 0xc0008fa8c0, 0xc000db4800)
        /usr/local/go/src/net/http/server.go:2867 +0xa3
    net/http.(*conn).serve(0xc00043e500, 0x2155d90, 0xc00087ff80)
        /usr/local/go/src/net/http/server.go:1932 +0x8cd
    created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2993 +0x39b
    level=warn ts=2021-11-30T02:10:54.095595359Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10000 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10000
    level=warn ts=2021-11-30T02:10:54.096047117Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10006 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10006
    level=warn ts=2021-11-30T02:10:54.097687309Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10007 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10007
    level=warn ts=2021-11-30T02:10:54.098111126Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10003 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10003
    level=warn ts=2021-11-30T02:10:54.098163667Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10005 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10005
    level=warn ts=2021-11-30T02:10:54.098497601Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10002 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10002
    level=warn ts=2021-11-30T02:10:54.098524609Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10001 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10001
    level=warn ts=2021-11-30T02:10:54.098719277Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10004 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10004
    level=warn ts=2021-11-30T02:10:54.305491706Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10010 after err: rpc error: code = Unavailable desc = Service Unavailable: HTTP status code 503; transport: received the unexpected content-type \"text/html\": rpc error: code = Unavailable desc = Service Unavailable: HTTP status code 503; transport: received the unexpected content-type \"text/html\"" address=10.150.15.92:10010
    level=warn ts=2021-11-30T02:10:59.099762426Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10000 after err: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10000
    level=warn ts=2021-11-30T02:10:59.099812437Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10006 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10006
    level=warn ts=2021-11-30T02:10:59.099842639Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10001 after err: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10001
    level=warn ts=2021-11-30T02:10:59.099923937Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10005 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10005
    level=warn ts=2021-11-30T02:10:59.099929147Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10004 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10004
    level=warn ts=2021-11-30T02:10:59.100016306Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from 10.150.15.92:10003 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=10.150.15.92:10003

Anything else we need to know:
@hanjm
Copy link
Member

hanjm commented Nov 30, 2021 •
edited
Loading

Seems log not full display, could you provider more log about panc stack. it should start with "panic: xxx". Thanks.

@gmintoco
Copy link
Contributor Author

Hi @hanjm here is the output from the logs:

    2021/11/30 04:02:26 http: panic serving 10.100.196.39:51262: runtime error: invalid memory address or nil pointer dereference
    goroutine 24149 [running]:
    net/http.(*conn).serve.func1(0xc0001a80a0)
        /usr/local/go/src/net/http/server.go:1804 +0x153
    panic(0x1b54060, 0x2f4a640)
        /usr/local/go/src/runtime/panic.go:971 +0x499
    github.com/thanos-io/thanos/pkg/api/query.(*QueryAPI).stores(0xc00030fba0, 0xc000fffb00, 0x0, 0x0, 0x0, 0x7000105, 0x0, 0xffffffffffffffff)
        /home/circleci/project/pkg/api/query/v1.go:717 +0xcb
    github.com/thanos-io/thanos/pkg/api.GetInstr.func1.1(0x2148fe0, 0xc000380d90, 0xc000fffb00)
        /home/circleci/project/pkg/api/api.go:211 +0x42
    net/http.HandlerFunc.ServeHTTP(0xc0007366d8, 0x2148fe0, 0xc000380d90, 0xc000fffb00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1(0x2148fe0, 0xc000380d90, 0xc000fffa00)
        /home/circleci/project/pkg/server/http/middleware/request_id.go:40 +0x20c
    net/http.HandlerFunc.ServeHTTP(0xc0007366f0, 0x2148fe0, 0xc000380d90, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x214c400, 0xc0000493e0, 0xc000fffa00)
        /home/circleci/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x299
    net/http.HandlerFunc.ServeHTTP(0xc00073b260, 0x214c400, 0xc0000493e0, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/thanos-io/thanos/pkg/logging.(*HTTPServerMiddleware).HTTPMiddleware.func1(0x214c400, 0xc0000493e0, 0xc000fffa00)
        /home/circleci/project/pkg/logging/http.go:68 +0x399
    net/http.HandlerFunc.ServeHTTP(0xc00073b290, 0x214c400, 0xc0000493e0, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/thanos-io/thanos/pkg/extprom/http.(*defaultInstrumentationMiddleware).NewHandler.func1(0x7fc8085d7410, 0xc0008aff90, 0xc000fffa00)
        /home/circleci/project/pkg/extprom/http/instrument_server.go:108 +0x10f
    net/http.HandlerFunc.ServeHTTP(0xc00073b350, 0x7fc8085d7410, 0xc0008aff90, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x7fc8085d7410, 0xc0008aff40, 0xc000fffa00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:198 +0xee
    net/http.HandlerFunc.ServeHTTP(0xc00073b650, 0x7fc8085d7410, 0xc0008aff40, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x7fc8085d7410, 0xc0008afef0, 0xc000fffa00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101 +0xdf
    net/http.HandlerFunc.ServeHTTP(0xc00073b8c0, 0x7fc8085d7410, 0xc0008afef0, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1(0x214e860, 0xc0007b6700, 0xc000fffa00)
        /home/circleci/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:165 +0xee
    net/http.HandlerFunc.ServeHTTP(0xc00073bbc0, 0x214e860, 0xc0007b6700, 0xc000fffa00)
        /usr/local/go/src/net/http/server.go:2049 +0x44
    github.com/thanos-io/thanos/pkg/tracing.HTTPMiddleware.func1(0x214e860, 0xc0007b6700, 0xc000fff700)
        /home/circleci/project/pkg/tracing/http.go:46 +0x54c
    github.com/prometheus/common/route.(*Router).handle.func1(0x214e860, 0xc0007b6700, 0xc000ffe400, 0x0, 0x0, 0x0)
        /home/circleci/go/pkg/mod/github.com/prometheus/common@v0.30.0/route/route.go:83 +0x27f
    github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000680d20, 0x214e860, 0xc0007b6700, 0xc000ffe400)
        /home/circleci/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0xc7e
    github.com/prometheus/common/route.(*Router).ServeHTTP(0xc00007f2c0, 0x214e860, 0xc0007b6700, 0xc000ffe400)
        /home/circleci/go/pkg/mod/github.com/prometheus/common@v0.30.0/route/route.go:121 +0x4c
    net/http.(*ServeMux).ServeHTTP(0xc00045d600, 0x214e860, 0xc0007b6700, 0xc000ffe400)
        /usr/local/go/src/net/http/server.go:2428 +0x1ad
    net/http.serverHandler.ServeHTTP(0xc0007b6000, 0x214e860, 0xc0007b6700, 0xc000ffe400)
        /usr/local/go/src/net/http/server.go:2867 +0xa3
    net/http.(*conn).serve(0xc0001a80a0, 0x2155c70, 0xc000cbc240)
        /usr/local/go/src/net/http/server.go:1932 +0x8cd
    created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2993 +0x39b

@hanjm
Copy link
Member

hanjm commented Nov 30, 2021

Same issue with #4777. It has fixed by #4805. Could you try the v0.24.0-rc.0? https://github.com/thanos-io/thanos/releases/tag/v0.24.0-rc.0

@gmintoco
Copy link
Contributor Author

gmintoco commented Dec 1, 2021

Hi @hanjm working now - thanks a lot :)

@gmintoco gmintoco closed this as completed Dec 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs-more-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hanjm @gmintoco