Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in Bucket Series #7885

Merged
merged 4 commits into from
Nov 7, 2024
Merged

Conversation

fpetkovski
Copy link
Contributor

@fpetkovski fpetkovski commented Nov 5, 2024

Applies the fix described in #7883.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Verification

Applies the fix described in thanos-io#7883.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
@@ -1691,7 +1691,9 @@ func (s *BucketStore) Series(req *storepb.SeriesRequest, seriesSrv storepb.Store
// Merge the sub-results from each selected block.
tracing.DoInSpan(ctx, "bucket_store_merge_all", func(ctx context.Context) {
begin := time.Now()
set := NewResponseDeduplicator(NewProxyResponseLoserTree(respSets...))
lt := NewProxyResponseLoserTree(respSets...)
defer lt.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to call Close() outside of this func(context.Context) because before Flush() we could still hold onto these sets?

Copy link
Contributor Author

@fpetkovski fpetkovski Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is a good point. Should we move lt := NewProxyResponseLoserTree(respSets...) outside of the span function as well, and we can defer a close right after it?

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
@harry671003
Copy link
Contributor

We also noticed another issue in Cortex even after cherry-picking this change. The pendingReaders for some blocks are not decremented correctly.

Store-Gateways were not able to sync blocks and was stuck at:

b.pendingReaders.Wait()

GiedriusS
GiedriusS previously approved these changes Nov 6, 2024
saswatamcode
saswatamcode previously approved these changes Nov 6, 2024
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
@fpetkovski fpetkovski merged commit 79593cb into thanos-io:main Nov 7, 2024
22 checks passed
@yeya24
Copy link
Contributor

yeya24 commented Nov 7, 2024

Do we want to address #7885 (comment) separately?
This seems still an issue.

@fpetkovski
Copy link
Contributor Author

Yes, will take a separate look 👍

GiedriusS pushed a commit to vinted/thanos that referenced this pull request Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants