query-frontend: cancel stream to scheduler when streaming failed #3302
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pstibrany
reviewed
Oct 25, 2022
There was a problem hiding this comment.
Looks good. I have one comment, and a proposal for a test:
https://gist.github.com/pstibrany/7a4384aee0b8389475ca7dd34af58410#file-frontend_test-go-L437
|
thanks for the review and the test @pstibrany. I think I've incorporated the changes you requested and the test too. Can you take another look? |
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
dimitarvdimitrov
force-pushed
the
dimitar/fix-query-frontend-grpc-stream-closure
branch
from
6e8b971 to
8076883
Compare
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
dimitarvdimitrov
force-pushed
the
dimitar/fix-query-frontend-grpc-stream-closure
branch
from
8076883 to
362838c
Compare
pracucci
approved these changes
Oct 26, 2022
pkg/frontend/v2/frontend_test.go
Outdated
| require.Equal(t, 1, checkStreamGoroutines()) | ||
| } | ||
|
|
||
| const createdBy = "created by google.golang.org/grpc.newClientStreamWithParams" |
There was a problem hiding this comment.
[nit] Move this const inside checkStreamGoroutines().
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Nov 4, 2022
…fana#3302) * query-frontend: cancel stream to scheduler when streaming failed Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Add changelog entry Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Add test for proper stream cleanup Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Simplify checkStreamGoroutines Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Nov 4, 2022
…fana#3302) * query-frontend: cancel stream to scheduler when streaming failed Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Add changelog entry Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Add test for proper stream cleanup Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * Simplify checkStreamGoroutines Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Dimitar Dimitrov dimitar.dimitrov@grafana.com
What this PR does
discovered by @pstibrany:
makes sure that the query-frontend properly terminates gRPC streams to the query-scheduler after the scheduler closes the connections on them
without this there was a goroutine and memory leak. In the attached screenshot you can see on the left side of each graph GEM running without this fix and on the right side of each graph GEM with this fix.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]