[release-2.9] Upgrade to thanos 0.32.3 #84
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: junot <junotxiang@kubesphere.io>
* Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in>
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
docs: mismatch in changelog
Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
…s-io#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
Signed-off-by: naveadkazi <navead@carwale.com>
Add CarTrade Tech as new adopter
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* store: add streamed snappy encoding for postings list
We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:
```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
flat flat% sum% cum cum%
0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1
2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode
```
This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.
Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.
Benchmark data:
```
name time/op
PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4%
name alloc/op
PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0%
name allocs/op
PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0%
```
Compression ratios are still the same like previously:
```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN TestDiffVarintCodec/snappy/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44498 bytes
postings_codec_test.go:81: ratio: 0.012
=== RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44670 bytes
postings_codec_test.go:81: ratio: 0.012
```
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: clean up postings code
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: fix estimation
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: use buffer.Bytes()
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store/postings_codec: reuse extgrpc compressors/decompressors
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: add item
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: clean up whitespace
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
…-heap-less-func Reuse buffers for label comparison
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Signed-off-by: Philip Gough <philip.p.gough@gmail.com>
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Author: Subbarao Meduri <smeduri@redhat.com> v0.32.3
Signed-off-by: Subbarao Meduri <smeduri@redhat.com>
subbarao-meduri
changed the title
douglascamata
approved these changes
Sep 20, 2023
|
/retest-required |
3 similar comments
|
/retest-required |
|
/retest-required |
|
/retest-required |
|
/test images |
saswatamcode
approved these changes
Sep 21, 2023
There was a problem hiding this comment.
What's the motive for removing this file? Seems like it is useful. 🙂
There was a problem hiding this comment.
I believe the instructions are dated 🙂
I used these instructions for doing the merge:
https://stackoverflow.com/questions/173919/is-there-a-theirs-version-of-git-merge-s-ours
(See section A possible and tested solution for merging branchB into our checked-out branchA:)
I will need to script this (separate PR) so it can be reused going forward.
| @@ -1,10 +0,0 @@ | |||
| sonar.projectKey=open-cluster-management_thanos | |||
There was a problem hiding this comment.
Is this from some previously used CI?
There was a problem hiding this comment.
Yes, we no longer run sonarCloud on maintenance forks.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: douglascamata, saswatamcode, subbarao-meduri The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@saswatamcode - docs / Documentation check (pull_request) seem to fail frequently due to unreachable URLs. Do you see the same behavior upstream too? Curious what it takes to fix this so it passes consistently. |
|
/retest-required |
|
Yup, the docs check fails due to lightstep.com no longer being available (changed to a servicenow link). I fixed it on main, but not on release branches. |
|
/retest-required |
1 similar comment
|
/retest-required |
|
@subbarao-meduri: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Merging as OpenShift CI problems are not resolved even after 24 ours. Given e2e tests have passed, we have high confidence. |
subbarao-meduri
added a commit
that referenced
this pull request
Nov 9, 2023
This reverts commit 8727d9f. Signed-off-by: Subbarao Meduri <smeduri@redhat.com>
|
/test pr-image-mirror |
|
/test images |
subbarao-meduri
added a commit
to subbarao-meduri/thanos
that referenced
this pull request
Nov 10, 2023
This reverts commit c2dccb9. Signed-off-by: Subbarao Meduri <smeduri@redhat.com>
subbarao-meduri
added a commit
that referenced
this pull request
Nov 10, 2023
* Revert "upgrade golang/x/net to 0.17.0 (#92)" This reverts commit 1a7d05c. Signed-off-by: Subbarao Meduri <smeduri@redhat.com> * Revert "Revert "[release-2.9] Upgrade to thanos 0.32.3 (#84)"" This reverts commit c2dccb9. Signed-off-by: Subbarao Meduri <smeduri@redhat.com> * Revert "Revert "upgrade golang/x/net to 0.17.0 (#85)"" This reverts commit 8214b23. Signed-off-by: Subbarao Meduri <smeduri@redhat.com> * Revert "Revert "update busybox version (#91)"" This reverts commit 485c4f7. Signed-off-by: Subbarao Meduri <smeduri@redhat.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Verification