Proxy: Query goroutine leak when store.response-timeout is set
#7618
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cincinnat
force-pushed
the
query-goroutine-leak
branch
4 times, most recently
from
910a242
to
a4b9301
Compare
saswatamcode
approved these changes
Aug 11, 2024
|
@cincinnat could you kindly rebase on latest main? We had a CI issue, which seems to fixed. Want to merge this on green 🙂 |
time.AfterFunc() returns a time.Timer object whose C field is nil, accroding to the documentation. A goroutine blocks forever on reading from a `nil` channel, leading to a goroutine leak on random slow queries. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>
cincinnat
force-pushed
the
query-goroutine-leak
branch
from
a4b9301
to
d23ca27
Compare
saswatamcode
approved these changes
Aug 13, 2024
saswatamcode
pushed a commit
to saswatamcode/thanos
that referenced
this pull request
Aug 13, 2024
…nos-io#7618) time.AfterFunc() returns a time.Timer object whose C field is nil, accroding to the documentation. A goroutine blocks forever on reading from a `nil` channel, leading to a goroutine leak on random slow queries. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>
saswatamcode
added a commit
that referenced
this pull request
Aug 13, 2024
* Proxy: Query goroutine leak when `store.response-timeout` is set (#7618) time.AfterFunc() returns a time.Timer object whose C field is nil, accroding to the documentation. A goroutine blocks forever on reading from a `nil` channel, leading to a goroutine leak on random slow queries. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * pkg/clientconfig: fix TLS configs with only CA (#7634) 065e3dd introduced a regression: TLS configurations for Thanos Ruler query and alerting with only a CA file failed to load. For instance, the following snippet is a valid query configuration: ``` - static_configs: - prometheus.example.com:9090 scheme: https http_config: tls_config: ca_file: /etc/ssl/cert.pem ``` The test fixtures (CA, certificate and key files) are copied from prometheus/common and are valid until 2072. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Cut patch release v0.36.1 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix failing e2e test (#7620) Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Co-authored-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Harry John <johrry@amazon.com>
saswatamcode
added a commit
that referenced
this pull request
Aug 14, 2024
* CHANGELOG: Mark 0.36 as in progress Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Cut release candidate v0.36.0-rc.0 (#7490) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Cut release candidate 0.36.0 rc.1 (#7510) * *: fix server grpc histograms (#7493) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Close endpoints after the gRPC server has terminated (#7509) Endpoints are currently closed as soon as we receive a SIGTERM or SIGINT. This causes in-flight queries to get cancelled since outgoing connections get closed instantly. This commit moves the endpoints.Close call after the grpc server shutdown to make sure connections are available as long as the server is running. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Cut release candidate v0.36.0-rc.1 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * Cut release v0.36.0 (#7578) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Cut patch release `v0.36.1` (#7636) * Proxy: Query goroutine leak when `store.response-timeout` is set (#7618) time.AfterFunc() returns a time.Timer object whose C field is nil, accroding to the documentation. A goroutine blocks forever on reading from a `nil` channel, leading to a goroutine leak on random slow queries. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * pkg/clientconfig: fix TLS configs with only CA (#7634) 065e3dd introduced a regression: TLS configurations for Thanos Ruler query and alerting with only a CA file failed to load. For instance, the following snippet is a valid query configuration: ``` - static_configs: - prometheus.example.com:9090 scheme: https http_config: tls_config: ca_file: /etc/ssl/cert.pem ``` The test fixtures (CA, certificate and key files) are copied from prometheus/common and are valid until 2072. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Cut patch release v0.36.1 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix failing e2e test (#7620) Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Co-authored-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Harry John <johrry@amazon.com> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Co-authored-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Harry John <johrry@amazon.com>
hczhu-db
pushed a commit
to databricks/thanos
that referenced
this pull request
Aug 22, 2024
* Proxy: Query goroutine leak when `store.response-timeout` is set (thanos-io#7618) time.AfterFunc() returns a time.Timer object whose C field is nil, accroding to the documentation. A goroutine blocks forever on reading from a `nil` channel, leading to a goroutine leak on random slow queries. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * pkg/clientconfig: fix TLS configs with only CA (thanos-io#7634) 065e3dd introduced a regression: TLS configurations for Thanos Ruler query and alerting with only a CA file failed to load. For instance, the following snippet is a valid query configuration: ``` - static_configs: - prometheus.example.com:9090 scheme: https http_config: tls_config: ca_file: /etc/ssl/cert.pem ``` The test fixtures (CA, certificate and key files) are copied from prometheus/common and are valid until 2072. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Cut patch release v0.36.1 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix failing e2e test (thanos-io#7620) Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com> Co-authored-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Harry John <johrry@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
time.AfterFunc()returns atime.Timerobject whoseCfield is nil, accroding to the documentation. A goroutine blocks forever on reading from anilchannel, leading to a goroutine leak on random slow queries for Thanos.This goroutine leak would be most apparent for busy services with
query.promql-engine=thanos, when grouroutins tend to stuck in batches, thanks to the wide usage ofsync.Onceby the engine.Changes
Verification