Compaction TODO metrics have too high a cardinality #5841

SuperQ · 2022-10-31T12:05:32Z

Thanos, Prometheus and Golang version used:

Thanos v0.28.1

Object Storage Provider:

N/A

What happened:

The metrics thanos_compact_todo_compactions, thanos_compact_todo_downsample_blocks, etc include a group label that can make the cardinality of the metric reach thousands in large deployments. This label doesn't provide a lot of value.

What you expected to happen:

Remove the group label from compaction TODO metirics.

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

The text was updated successfully, but these errors were encountered:

yeya24 · 2022-10-31T15:24:29Z

Almost all the compaction metrics have the label group, right? Why it is an issue for the TODO metrics?

SuperQ · 2022-10-31T15:36:36Z

After digging through the metrics we have, I discovered it's actually a problem for all of the metrics that have group on them.

I have a WIP change to remove group label from all of the compactor metrics if people find this acceptable.

Akshit42-hue · 2022-10-31T16:55:14Z

Working on it :)

yeya24 · 2022-10-31T17:11:08Z

Hey @Akshit42-hue, I think we still need some more discussions to see whether we want to do it or not.

yeya24 · 2022-11-01T02:27:25Z

I have a WIP change to remove group label from all of the compactor metrics if people find this acceptable.

I think it is still valuable to have a label to identify each compaction group so that we can identify some key metrics like failutres.
Unfortunately the current label group is not a good identifier cause I cannot tell which cluster/tenant it is from the given label hash.

SuperQ · 2022-11-15T06:01:01Z

Metrics should identify when there are compaction failures worth investigating. The actual identity of what is failing can be determined from the logs.

yeya24 · 2023-01-17T02:37:18Z

Let's do it. I think we should remove this label as the hash doesn't mean anything. However, I am wondering if we should still keep the downsampling information as another label.

Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Add a `resolution` label to the compaction counters. Fixes: thanos-io#5841 Signed-off-by: SuperQ <superq@gmail.com>

SuperQ · 2023-01-17T11:06:55Z

Downsample resolution sounds like a useful label.

Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Add a `resolution` label to the compaction counters. Fixes: thanos-io#5841 Signed-off-by: SuperQ <superq@gmail.com>

Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: thanos-io#5841 Signed-off-by: SuperQ <superq@gmail.com>

* Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb. The added t.Logf() showed that is the problem is that with a lower bytes limit, it might hit the series or chunks part first. I have bumped the bytes limit. I calculated the new bytes limit by checking how much bytes are allocated before sending the last chunk. I have also noticed that one block is created without a delay. Update it so that it would be like the others. Include objstore@main update with https://github.com/thanos-io/objstore/pull/62/files so that Iter() would always return an error on a timeout. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * pkg/reloader: use watchInterval timeout for initial apply (#6519) * pkg/reloader: use watchInterval timeout for initial apply Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * changelog Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * use distinct context for initial sync Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> --------- Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * Standardize index cache metrics (#6523) * Add histogram metrics for index cache item size (#6528) * add histogram metrics for index cache item size Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Upgrade objstore (#6507) * Upgrade objstore Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update with the latest APIs Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> --------- Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Make compact lifecycle more flexible to be overridden for sharded compaction (#5964) * Updates busybox SHA (#6534) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * BucketedBytes to buffer byte slices when decoding postings from cache (#6531) * BucketedBytes to buffer byte slices when deconding postings from cache Signed-off-by: Alan Protasio <alanprot@gmail.com> * Fixing Lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Shipper: change upload compacted type from bool to a function (#6526) * change shipper upload compacted type from bool to a function Signed-off-by: Ben Ye <benye@amazon.com> * add default to false Signed-off-by: Ben Ye <benye@amazon.com> * reset uploadedCompacted to 0 Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Query: Forward tenant information via StoreAPI (#6530) * Querier: Forward tenant information downstream With this commit we attach tenant information to each query request and forward it via the StoreAPI to any downstream Store Gateways and Queriers. We add the following command lines options which mimics the tenant functionality in Receive. The options are currently hidden, as they provide no real functionality yet. This will come in future steps. --query.tenant-header --query.default-tenant --query.tenant-certificate Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Receive: Use CertificateField from Tenancy pkg These consts are now defined in the Tenancy package, so we should use those instead. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Deduplicate matchers in posting group (#6532) * deduplicate matchers in posting group Signed-off-by: Ben Ye <benye@amazon.com> * don't use map for deduplication Signed-off-by: Ben Ye <benye@amazon.com> * lint Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * fi…

* fix 0.31 changelog (#6278) Signed-off-by: junot <junotxiang@kubesphere.io> * Query: Switch Multiple Engines (#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with http…

Compaction dowsnample metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a resolution label to reduce cardinality. Fixes: thanos-io#5841 Signed-off-by: SuperQ <superq@gmail.com>

Compaction dowsnample metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a resolution label to reduce cardinality. Fixes: #5841 Signed-off-by: SuperQ <superq@gmail.com>

…o#7283) Compaction dowsnample metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a resolution label to reduce cardinality. Fixes: thanos-io#5841 Signed-off-by: SuperQ <superq@gmail.com>

* CI: increase e2e test parallelism (#6875) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * compact: hook nodownsamplemarkfilter into filters chain (#6893) We have a NoDownsampleMarkFilter that we were not using before in the compactor for some reason. Hook it into the filters chain if downsampling is enabled and then trim matching ULIDs from the downsampling process. Add a test to cover this scenario. Fixes https://github.com/thanos-io/thanos/issues/6179. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e/compact: fix assertion values (#6909) I think the original values were simply too small there - if the runner is slow then it can see smaller metrics values. If the runner works well then the values can go to these bigger values and then the test fails. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * exposes asyncOperationProcessor (#6908) expose struct fix lint Signed-off-by: Ben Ye <benye@amazon.com> * Removed the gitpod from the contribution.md (#6907) Signed-off-by: MeenuyD <meenu.coninja@gmail.com> * .circleci: cache tool deps (#6910) I have noticed that downloading tool deps takes ~5 min in CI each time. We should be able to also cache those tool deps. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * MAINTAINERS.md: add myself as maintainer (#6911) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * blog: Introducing ThanosCon in Paris KubeCon EU 2024 (#6912) * blog: Introducing ThanosCon in Paris KubeCon EU 2024 Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Add image Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * make docs Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Apply suggestions Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Apply suggestions Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Add link to slack channel Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update 2023-20-11-thanoscon.md (typo) (#6915) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added info about ThanosCon to website and Readme. (#6917) * Added info about ThanosCon to website and Readme. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update website/layouts/index.html Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Objstore: Bump Objstore for Azure Workload Identity support (#6891) * Bump objstore for Azure Workload Identity support Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * make check-docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Add changelog entry Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Update Azure client docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * make check-docs Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Move changelog entry to 0.33.0 release Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> * Move changelog entry Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> --------- Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * update compactor backlog doc for checking halt (#6906) Signed-off-by: Ben Ye <benye@amazon.com> * Update LabyrintLabs logo Signed-off-by: Martin Hauskrecht <hauskrechtmartin@gmail.com> * Support reload using signal (#6453) * Support reload using signal Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Add --reloader.method option to sidecar This option allows to tell the sidecar to send a SIGHUP signal to the monitored process to reload its configuration instead of the default HTTP-based method. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Update docs and CHANGELOG.md Signed-off-by: Simon Pasquier <spasquie@redhat.com> --------- Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Support float histogram in store gateway (#6925) * support float histogram in store gateway Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> * update prompb Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * chore: add truelayer to the adopters (#6933) Signed-off-by: Matteo Martellini <matteo.martellini@truelayer.com> * Fetcher: Add a BlockIDsFetcher Interface to BaseFetcher (#6902) * add BlockIDsFetcher to BaseFetcher Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix lint Signed-off-by: Wen Xu <wenxuamz@amazon.com> * use chan in the interface method to accept active block ids Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix comments Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix lint Signed-off-by: Wen Xu <wenxuamz@amazon.com> * add description of active and parital blocks and modify changelog Signed-off-by: Wen Xu <wenxuamz@amazon.com> * fix interface description Signed-off-by: Wen Xu <wenxuamz@amazon.com> * remove entry in changelog Signed-off-by: Wen Xu <wenxuamz@amazon.com> --------- Signed-off-by: Wen Xu <wenxuamz@amazon.com> * Update hugo file (#6927) * Update hugo file #6889 Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Version 0.33 removed Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Updates busybox SHA (#6897) (#6937) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix hotstar logo (#6938) * Fix hotstar logo Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Delete website/.hugo_build.lock Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * errutil: fix deadlock Fix deadlock in the Add() function - it accidentally was calling itself recursively. This struct wraps a `MultiError` so we need to call `Add()` on that `MultiError` inside. This deadlock can manifest in Receive - million+ leaking goroutines, all stuck on the MultiTSDB lock. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * update runtime/CloseWithErrCapture comment (#6952) * update runtime/CloseWithErrCapture comment Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> * fix lint Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> --------- Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com> * Added a new flag for maximum retention bytes for thanos (#6944) * changes Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * Changed Int64 to BytesVar Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * Updated Changelog for new flag in receive Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * fix store gateway span name (#6953) Signed-off-by: Ben Ye <benye@amazon.com> * Upgrade prometheus to latest main (#6950) * upgrade prometheus to latest main Signed-off-by: Ben Ye <benye@amazon.com> * upgrade again Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Improve filtered index cache filtering (#6955) * improve filtered index cache filtering Signed-off-by: Ben Ye <benye@amazon.com> * make inline Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * store: fix misc label_value issues related to external labels (#6879) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * receive: fix adding fields to logger Fix adding fields to the logger because otherwise it looks like "[tenant foo]: (UNDEFINED)". Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Copy labels coming from remote engines (#6957) * Copy labels coming from remote engines When running in distributed mode, the remote engine will use an unsafe cast from ZLabels to Prometheus labels to avoid making new allocations. This makes it hard to use the new gRPC shared buffer pool for receiving and decompressing messages since memory gets retained beyond the scope of a Recv() call. This commit removes the unsafe cast and makes an explicit memory copy of received series labels. Since remote queries are already aggregated series, the amount of data we receive should be small anyway, and the copies on average should have a small impact. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use clone on strings Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add tracing index cache (#6954) * add tracing index cache Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Change order of log for finished compacting blocks (#6966) Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Update MAINTAINERS.md (#6967) Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Add `keep_firing_for` support to alerting rule (#6943) * Add keep_firing_for to alert rules Signed-off-by: JHeil <jheil@coveo.com> * Updating changelog Signed-off-by: JHeil <jheil@coveo.com> * Re-order keepFiringFor at the end of the struct Signed-off-by: JHeil <jheil@coveo.com> --------- Signed-off-by: JHeil <jheil@coveo.com> Signed-off-by: JHeil <52839010+JHeilCoveo@users.noreply.github.com> * Use registerer for gRPC opts and endpoints (#6971) The EndpointSet and StoreGRPC options accept concrete *prometheus.Registry type. This makes it hard to create multiple instances of them because they will try to register the same metrics and the registry will panic. This commit changes the type to a prometheus.Registerer which allows clients to inject a prefixed registry and get metrics for each instance. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * receive: upload compacted blocks if OOO enabled (#6974) * Add streaming series limit at block series client (#6972) * add series limit that is applied when streaming using block series client Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * add unit tests Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * fix comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Updated the defaultChecked property in checkbox element (#6976) Signed-off-by: lpreethvika-samsung <l.preethvika@samsung.com> * Update prometheus to d0c2d9c (#6978) * Update prometheus to d0c2d9c This commit updates Prometheus to commit d0c2d9c which contains the loser-tree based postings merge. Note that we cannot update to latest main since there is currently a bug in PromQL when querying native histograms introduced by https://github.com/prometheus/prometheus/pull/13276. This issue should be fixed by https://github.com/prometheus/prometheus/pull/13289. This commit also updates the Thanos PromQL engine to latest main. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix handler_test.go Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix manager_test.go Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use empty registry for file discovery Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6982) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Updates busybox SHA (#6982) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * VERSION: Bump version for release Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * rule: native histogram support (#6390) * Added native histogram support for ruler Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Formatted imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Formated imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed receiver type Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix for rebase Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added docs for query endpoints differences Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed comments and naming Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * made HTTPConfig optional Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * made HTTPConfig optional Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted and added check Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixes from comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * renamed queryconfig to clientconfig Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * common prepareEndpointSet Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed lint Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed sidecar Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * mdox ignore checking twitter urls (#7001) Signed-off-by: Ben Ye <benye@amazon.com> * Updated Grofers logo (#7006) Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * TraceID : Fetching TraceID (#6973) * docs: add promcon 2023 thanos talks Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Added website page for companies who offer consultancy and enterprise… (#7000) * Added website page for companies who offer consultancy and enterprise support for Thanos Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * adopters.yml revert Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * retrigger checks Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * added a new line in welcome.md Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * retrigger checks Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * Lazy downloaded index header (#6984) * lazy downloaded index header Signed-off-by: Ben Ye <benye@amazon.com> * update tests Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * tests: use remote write in query frontend tests (#6998) * query-frontend: Added support of auto_discovery for memcached (#7004) * query-frontend: Added support of auto_discovery for memcached Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * adjustments to build on main branch Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * CHANGELOG.md Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * typo fixed Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * minor fixex after review Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> --------- Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * Fix URI encoding of strings Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * ui: enable partial response strategy by default Rebuild Signed-off-by: Vanshika <vanshikav928@gmail.com> * Added negative offset check for caching queries (#7011) Signed-off-by: pawarpranav83 <pawarpranav@gmail.com> Co-authored-by: pawarpranav83 <pawarpranav@gmail.com> * Update MAINTAINERS.md (Bartek's company) Should be done in January last year... Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * store: add chunksize tests to acceptance tests * add chunk size tests to acceptance tests * refactor acceptance tests slightly Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * added tasrie it services as support partner (#7023) * added tasrie it services as support partner Signed-off-by: Tasrie IT Services <info@tasrieit.com> * added the link in the list.html page Signed-off-by: Tasrie IT Services <info@tasrieit.com> * fixed the logo resolution to 190x70 Signed-off-by: Tasrie IT Services <info@tasrieit.com> * Remove extraneous text Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Tasrie IT Services <info@tasrieit.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Query Frontend: Add tenant label to metrics (#6887) This commit adds a tenant label to the HTTP metrics which are exported by the Query Frontend. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * tests: use remote write in query frontend tests (#7017) * resolved issue #7024 Signed-off-by: Pratham Agarwal <agarwalpratham1812@gmail.com> * QueryFrontend|Query: Create new arg to enable extended functions (#7028) * Adding new parameter for extended functions in querier Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com> * Adding new flag for QFE Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * improve argument passing in query side Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding e2e test for query Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * undoing uneeded changes Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing docs Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing e2e tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing backward compat test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * changes from CR Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * receive: respect forward timeout in http handler buckets (#7030) Forward timeout is what ultimately decides how long a HTTP request might take so it doesn't make sense to hard-code max value of 5. In this pull request I propose respecting the configured forward timeout and adding extra buckets if the default buckets don't cover it completely. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive: do not leak grpc connections (#7031) * receive: do not leak grpc connections Prevent a leak in gRPC connections by garbage collecting old ones when the hashring changes. For that purpose, I propose adding a `Nodes() string` method so that it would be possible to know what nodes do not exist in the hashring anymore. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive: change order of operations Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * fix multi-tenancy link (#7032) Signed-off-by: Alec Rajeev <ccarajeev342@gmail.com> * removing todo comments from query docs Signed-off-by: Harsh Pratap Singh <harshpratapsingh8210@gmail.com> * Query: add optional tenancy enforcement (#6756) * Query: add optional tenancy enforcement With this commit it's now possible to enable enforcement of tenancy. If tenancy is enabled, a tenant label will be added to queries based on the tenant information provided by the tenant header, and the tenant-label-name. The implementation for query APIs are done by using prom-label-proxy as library, while the implementation for non-query APIs are written from scratch. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Add changelog entry Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Query: Add non-default tenant testcase Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Test: make query a constant to make linter happy Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments - Remove empty lines - If multiple tenant matchers are found in the original query, we only replace the first one with the header provided tenant, and remove any subsequent ones. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments - `--enable-tenancy` -> `--enforce-tenancy` - Create `RewritePromQL` and `RewriteLabelMatchers` to clean up code in query api. Also move getLabelMatchers to tenancy pkg. - Use prom-label-proxys `EnforceMatchers` to rewrite labels on non-query APIs instead of own solution - Don't specifically handle `illegalLabelMatcherError` Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Re-arrage go.mod to make linter happy. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Address review comments Minor changes to CLI docs, code-comments and changelog. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Query: Add tenancy docs This commit adds documentation for the tenancy features. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Update docs/components/query.md Review comment Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Tests: unique docker env name for tenant tests (#7038) The e2e tests would occasionally fail due to non-unqiue docker environment names. With this commit the tests are environments are given unique names to avoid these failures. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * changed default 3m to 15m Signed-off-by: Player256 <dattucodes@gmail.com> * Fix docs (#7039) Fix docs post #6539 merge. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert "receive: upload compacted blocks if OOO enabled (#6974)" (#7053) This reverts commit 7b8eb86c0ff3b1144aa1cea392806afa40a4cdf8. Proper way to handle this is to disable vertical compaction. I am trying to add this functionality here: https://github.com/prometheus/prometheus/pull/13393 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [CHORE] adding thanos upload-snapshot command (#6884) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * fix docs error Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com> * cmd: make bucket upload command take lset from flags (#7059) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * CHANGELOG: mark 0.34 as in progress Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.0-rc.0 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * UI: Don't always force tracing (#7062) Forced tracing was.. Forced true always, even if the checkbox in the UI to enable tracing was not actually checked. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * go.mod: update Prometheus version (#7047) Update Prometheus version to include https://github.com/prometheus/prometheus/pull/13242 which is important for me - it unblocks further postings work. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Make RetryError and HaltError able to be fetched for root cause (#7043) * Make RetryError and HaltError able to be fetched for root cause Signed-off-by: Alex Le <leqiyue@amazon.com> * Added unit test Signed-off-by: Alex Le <leqiyue@amazon.com> * fix lint Signed-off-by: Alex Le <leqiyue@amazon.com> * fixed IsRetryError and IsHaltError functions Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> * receive: disable overlapping compaction Use the new TSDB flag to disable overlapping compaction to fix OOO samples handling in the Receive component. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CI: Ensure static react-app is checked in (#7063) * CI: Ensure static react-app is checked in With this commit the CI system should fail if changes to the react-app has been made without checking in the changes. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Add `react-app` as dependency `check-react-app` To ensure the react-app is rebuilt before checking for changes. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Receive: refactor handler for improved readability and organization (#6898) * [wip] First checkpoint Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Second checkpoint All tests passing, unit and e2e. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Small random refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add some useful trace tags Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Concurrent and traced local writes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve variable names in remote writes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename `newFanoutForward` function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * More refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting issue Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a quorum test with sloppy quorum Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Try to make retries work Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [wip] Checkpoint: wait group still hanging Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Some refactors Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add some commented code so I don't lose it Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Adapt tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove sloppy quorum code Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move some code around Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove even more leftover of sloppy quorum Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract a type to hold function params Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove unused struct field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove useless variable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove type that wasn't used enough Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Delete function to tighten up max buffered responses Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments to some functions Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix peer up check Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix size of replication tracking slices Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename context Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Don't do local writes concurrently Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove extra error logging Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix syntax after merge Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add missing methods to peersContainer Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix handler test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reset peers state on hashring changes Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Handle PR comment regarding waitgroup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Set span tags to help debug Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix concurrency issue We close the request as soon as quorum is reached and leave a few Go routines running to finish replication and so cleanups. This means that the context from the HTTP request is cancelled... which ends up also cancelling the pending replication requests. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix request ID middleware Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix `distributeTimeseriesToReplicas` comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract var with 1-indexed replication index Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rename methods in peersContainer interface Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make peerGroup `getConnection` check if peers are up Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove yet one more not useful log Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove logger from `h.sendWrites` Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Upgrade grpc to 1.57.2 (#7078) 1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2 2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail Signed-off-by: hanyuting8 <hytxidian@163.com> * Store: acceptance test for proxy store (#7084) * Add basic acceptance tests for proxy store * Fix bug where invalid requests got ignored because of partial response strategy Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Fix lazy postings with zero length (#7083) * fix lazy postings with zero length Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * unit tests Signed-off-by: Ben Ye <benye@amazon.com> * fix doc Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Store: fix label values edge case (#7082) If the requested label is an external label and we have series matchers we should only return results if the series matchers actually match a series. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.0-rc.1 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * receive: race condition in handler Close() when stopped early (#7087) Receiver hangs waiting for the HTTP Hander to shutdown if an error occurs before Handler is initialized. This might happen, for example, if the hashring is too small for a given replication factor. Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io> * receive: use async remote writing (#7045) * CHANGELOG: cut release 0.34 (#7095) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Stores: convert tests to not rely on slice labels (#7098) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * all: get rid of query pushdown to simplify query path (#7014) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Store: dont rely on slice labels continued Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Update prometheus/prometheus (#7096) * Update prometheus/prometheus This commit updates prometheus/prometheus to latest main (60b6266e). Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix file discovery Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * receive/handler: fix locking twice (#7112) Fix bug introduced in https://github.com/thanos-io/thanos/pull/6898: we were RLock()ing twice. This leads to a deadlock in some situations. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * fix minio store gateway err (#7114) Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> * receive/handler: do not double lock (#7124) markPeerUnavailable was always taking a lock and in one case we were calling it with a lock already taken. Fix this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * default to alertmanager v2 api (#7123) Signed-off-by: Jake Keeys <jake@keeys.org> * Receive: dont rely on slice labels (#7100) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Upgrade grpc to 1.57.2 (#7078) 1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2 2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail Signed-off-by: hanyuting8 <hytxidian@163.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * VERSION: cut release 0.34.1 Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * docs: update helm installation instruction the prometheus helm chart is a community maintained chart since a few years. With that, the old example pointed to an old chart and the provided example values aren't also working anymore. This update the documentation. Signed-off-by: Mario Constanti <github@constanti.de> * docs: fix link (#7129) The link has moved to another since Cisco bought Banzai Cloud. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: run make docs for helm installation instruction Signed-off-by: Mario Constanti <github@constanti.de> * Fixing log line for remote engine in debug mode (#7133) Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding new method on BucketedBytes to expose used memory (#7137) * Adding new method on bucketed bytes to expose used Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Removing interface, using RWMutex Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding InfraCloud as Enterprise support partner (#7141) * adding InfraCloud as Enterprise support partner Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * replaced svg file to match layout Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * added alt-text and horizontal image Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> --------- Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io> * docs: fix link (#7129) The link has moved to another since Cisco bought Banzai Cloud. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * cache: attach object storage hash to iter key (#6880) Attach object storage hash to the iter key so that it would be possible to reuse the same cache storage e.g. Redis for different buckets. Without this, the results are funny to say the least if you accidentally attempt to do that. Thus, let's add the hash to reduce the possibility of an accident for our users. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add support for extended promql functions in rule (#7105) Adds a flag to register the extended promql functions supported by the thanos query engine when running the rule component. This will allow rule config files containing query expressions with (xrate / xincrease / xdelta) to pass validation. This will only work if the query endpoint in use is running the thanos engine. Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Extended func support - doc update (#7161) * Add support for extended promql functions in rule Adds a flag to register the extended promql functions supported by the thanos query engine when running the rule component. This will allow rule config files containing query expressions with (xrate / xincrease / xdelta) to pass validation. This will only work if the query endpoint in use is running the thanos engine. Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Update rendered docs with added flag Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> --------- Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com> * Copy labels from remote instant queries (#7151) Similar to https://github.com/thanos-io/thanos/pull/6957, we should copy labels from remote instant queries so that memory does not get overwritten when processing series in a central engine. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/query_frontend: add tests for explain/analyze (#7160) Adding tests for explain/analyze with QFE. Will add fixes as separate PR. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cache/caching_bucket: add path to hash (#7158) Add path to the hash. This allows identifying difference instances by different config paths. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Allow using different listing strategies (#7134) * Allow using different listing strategies Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Expose flags for block list strategy Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Run make docs Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix whitespace Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache: implement the circuit breaker pattern for asynchronous set operations in the cache client (#7010) * Implement the circuit breaker pattern for asynchronous set operations in the cache client Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Add feature flag for circuitbreaker Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Sync docs Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Skip configuration validation if the circuit breaker is disabled Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Make lint happy Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Abstract the logic of the circuit breaker Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> --------- Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * queryfrontend: fix analysis after API changes Fix the analysis functionality with query-frontend after the recent changes. Added tests for this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * bugfix: lazy posting optimization with wrong cardinality for estimation (#7122) * bugfix: catch lazy posting optimization using wrong cardinality for estimation Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * *: properly treat native histogram deduplication in chunk series merger We have detected a problem in the chunk seriers merger where it will panic in case it encounters native histogram chunks. I am using thanos as a library for a project and wanted to use the penalty function to dedup blocks from Prometheus instances. Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Query UI: Add tenant box (#6867) * Query UI: Add tenant box With this commit as tenant box is added to the query UI. It can be used to specify which tenant to use when making a query. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Re-compile static react app Recompiles the static react app as now needed following: https://github.com/thanos-io/thanos/pull/6900 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Move changelog item to appropiate future release After merging it was under the 0.34 release. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Move query path tenancy proposal to done Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * receive: add support for globbing tenant specifiers (#7155) We want to be able to route all tenants which begin with certain letters to some receivers so we need to have some kind of globbing/regex support in the hashring. This PR adds that functionality. We've been using this in prod successfully. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * .circleci: bump setup_docker_version version (#7177) The current image is deprecated. See https://discuss.circleci.com/t/remote-docker-image-deprecations-and-eol-for-2024/50176. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Unhide distributed execution mode This commit exposes the distributed query execution mode to end-users by unhiding the flag used to toggle this feature. The commit also adds documentation on when the mode is appropriate to be used. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add changelog entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/querier: fix analysis merging (#7179) We were not merging analysis properly - mergo was overwriting data. Instead of using a whole library for this, just write two small functions and use them. Add test to cover this. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fix docs Fixes docs formatting and updates the distributed execution link to the done proposal. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Added platformengineers Signed-off-by: Munir Khakhi <munir@improwised.com> * Added platformengineers (#7181) Signed-off-by: Munir Khakhi <munir@improwised.com> * Bump google.golang.org/protobuf to v1.33.0 (#7191) This PR bumps the version of google.golang.org/protobuf to v1.33.0 fix a potential vulnerability in the protojson.Unmarhsl function [1] that can occure when unmarshaling a message with a protobuf value. Even if the function isn't used directly in Thanos it would be safer to just bump it directly. [1] https://pkg.go.dev/vuln/GO-2024-2611 Signed-off-by: Daniel Mellado <dmellado@redhat.com> * fix: make responsive support page and change size of platform-engineer logo fix: add anchor tag to all images Signed-off-by: Payal17122000 <raviyapayal17@gmail.com> * downsample: retry objstore related errors Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * downsample: retry objstore related errors Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * rule: do not turn off if resolving fails (#7192) Do not turn off Ruler if resolving fails. We can still (try to) evaluate rules even if Alertmanager is not available. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Query UI: Only show tenant box with enforcement on (#7186) With this commit we only show the tenant-ui box when enforcement of tenancy is on, as it is not needed otherwise. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Reloader: Add support for watching and decompressing Prometheus configuration directories (#7199) Signed-off-by: Daniel Hrabovcak <thespiritxiii@gmail.com> * [FEAT] Adding blog post (#7202) Signed-off-by: Helia Barroso <helia.barroso@hotmail.com> Co-authored-by: Helia Barroso <helia.barroso@hotmail.com> * storepb: make ServerAsClient channels unbuffered Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add support for TSDB selector in querier (#7200) * Add support for TSDB selector in querier This PR allows using the query distributed mode against a set of multi-tenant receivers as described in https://github.com/thanos-io/thanos/blob/main/docs/proposals-done/202301-distributed-query-execution.md#distributed-execution-against-receive-components. The feature is enabled by a selector.relabel-config flag in the Query component which allows it to select a subset of TSDBs to query based on their external labels. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry and fix docs Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add comments Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for MatchersForLabelSets Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix failing test Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use an unbuffered channel Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Change flag description Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove parameter from ServerAsClient Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Update thanos-io/promql-engine (#7215) * Update thanos-io/promql-engine This commit updates the promql-engine module to latest main and modifies to remote engine based on the breaking change. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fix lint Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * expose NewPromSeriesSet (#7214) Signed-off-by: Ben Ye <benye@amazon.com> * [CHORE] adding auto GOMEMLIMIT flag (#7223) Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * bump objstore package version to latest main (#7228) Signed-off-by: Ben Ye <benye@amazon.com> * Query-frontend: fix missing redis username config (#7224) * add username cfg to rueidis client Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * UI: Showing Block Size Stats (#7233) * feat(ui): added BlockSizeStats calculation to blocks page A block can have a list of contained files set in `.thanos.files`. If the `files` array is set, all referenced files with `size_bytes` set are counted: - sum of all `chunk/*` file sizes - size of index file - total size (sum of both) Shows statistics about the selected block in the block details view: - Total size of block - Size of index (and percentage of total) - Size of all chunks (and percentage of total) - Daily growth, based on total size and block duration Output is humanized up to Pebibytes and fixed to two decimal places; raw bytes are accessible through mouse over / title text. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * feat(ui): added aggregated BlockSizeStats to blocks row title Added total size of all blocks from a source to the row title, beneath the source name. The shown total size is humanized up to pebibytes and fixed to two decimal places; raw bytes value is accessible through mouse over / title text. The shown value will refresh with selected compaction levels, but doesn't take block filter into account. I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(docs): added UI block size PR to CHANGELOG.md Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(ui): removed comments Automatic code formatting duplicated some comments near import statements. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> --------- Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * Fix lazy expanded postings cache and bug of non equal matcher (#7220) * fix lazy expanded postings cache and bug of non equal matcher with non existent values Signed-off-by: Ben Ye <benye@amazon.com> * test case for remove keys noop Signed-off-by: Ben Ye <benye@amazon.com> * add promqlsmith fuzz test Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix go mod Signed-off-by: Ben Ye <benye@amazon.com> * rename test Signed-off-by: Ben Ye <benye@amazon.com> * fix series request timestamp Signed-off-by: Ben Ye <benye@amazon.com> * skip e2e test Signed-off-by: Ben Ye <benye@amazon.com> * handle non lazy expanded case Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump Prometheus to include new label regex optimization (#7232) * bump Prometheus version to include new label matcher regex value optimization Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * fix again Signed-off-by: Ben Ye <benye@amazon.com> * include latest fix Signed-off-by: Ben Ye <benye@amazon.com> * update go mod Signed-off-by: Ben Ye <benye@amazon.com> * fix explain test Signed-off-by: Ben Ye <benye@amazon.com> * fix test again Signed-off-by: Ben Ye <benye@amazon.com> * update again Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * fix tests so far Signed-off-by: Ben Ye <benye@amazon.com> * fix compactor tests Signed-off-by: Ben Ye <benye@amazon.com> * use own out of order chunk index Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * update docs for receive routing only with limits (#7241) Signed-off-by: Alec Rajeev <13004609+alecrajeev@users.noreply.github.com> * docs: add thanoscon 2024 talks (#7243) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * remove write method from Compactor interface (#7246) Signed-off-by: Ben Ye <benye@amazon.com> * change shipper to not overwrite all external labels (#7247) Signed-off-by: Ben Ye <benye@amazon.com> * fix(changelog): fix GOMEMLIMIT pull request reference Signed-off-by: roth-wine <philipp.roth@hetzner.com> * Update kakkoyun's affiliation (#7251) * Added Shield in adopters (#7254) * Added Shield in adopters Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> * Upload compatible image Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com> --------- Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com> Co-authored-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com> * Tracing: added missing sampler types (#7231) * added missing sampler types Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * added changelog entry Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * fixed changelog entry Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * Fixed changelog entry conflict Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> --------- Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com> * fix query_test when --race enabled (#7258) Signed-off-by: Yi Jin <yi.jin@databricks.com> * go.mod: bump promql-engine (#7263) Bump promql-engine version to include samples counting. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fix 7244 error targets page (#7245) * added UNKNOWN to TargetHealth_value at targets proto Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> * added TargetHealth_value UNKNOWN to rpc.pb.go Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> --------- Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com> * clarify documentation around selecor.relabel-config option Signed-off-by: notafile <nota@notafile.com> * clarify writing around sharding Signed-off-by: notafile <nota@notafile.com> * don't halt compaction due to overlapping sources when vertical compaction is enabled (#7225) Signed-off-by: Ben Ye <benye@amazon.com> * Propagate the query plan * Serialize the plan for remote executions latest engine Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Propagate marshaled plan and introduce optimizer Propagating the query plan in the remote engine requests and introduce new SetProjectionColumns optimizer Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing passing down of plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * go mod tidy Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * avoid panics Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * delete dev file Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * undo small refactor Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * improve test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> generating protos Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> fixing v1 Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> delete unused method Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Set projection labels after distributing queries * removing second precision engine, upstream already truncates Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Passing the plan along as the query in remote executions Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Using proper constructors passing the query plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Refactor query creation from plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fallback in case we cant use plan Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * refactor, add tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Refactor to method Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * last nits Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * api/ui: show peak/total samples in analyze (#7269) Show the new peak/total fields in analyze output next to each operator. Add tooltips to explain what is the meaning of each field. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive/handler: implement tenant label splitting (#7256) * receive/handler: implement tenant label splitting Implement splitting incoming HTTP requests along some label inside of the timeseries themselves. This functionality is useful when you have one big application exposing lots of series and, for instance, you have a label `team` that identifies different owners of metrics in that application. Then using this you can use that `team` label to have different tenants in Thanos. Only negative thing that I could spot is that if after splitting one of the requests fails then that code is used for all tenants and that skews the Receiver metrics a little bit. I think that can be left as a TODO task. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * test/e2e: add more receiver tests Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * thanos/receive: note that splitting takes precendence over HTTP Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * thanos/receive: fix typo Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * query: fixing dedup iterator when working on mixed sample types (#7271) * query: fixing dedup iterator when working on mixed sample types There was a panic in case the dedupiterator worked on two chunks with both Native Histograms and Float (XOR encoded). Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Adding changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing lint Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * removing comments Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing repro test case Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing initialization Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing changelog Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * adding header to new file Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * using t.Run Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing ordering of samples in tests Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * change the reflect package to an unsafe package (#7143) - as 'reflect.String.Header' is deprecated, it is replaced with an unsafe package. Signed-off-by: Youngjun <yj.yoo@okestro.com> * Receive: fix issue-7248 with parallel receive_forward (#7267) * Receive: fix issue-7248 by introducing a worker pool Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix unit test bug Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix CLI flags not pass into the receive handler Signed-off-by: Yi Jin <yi.jin@databricks.com> * address comments Signed-off-by: Yi Jin <yi.jin@databricks.com> * init context in constructor Signed-off-by: Yi Jin <yi.jin@databricks.com> --------- Signed-off-by: Yi Jin <yi.jin@databricks.com> * Fix incorrect comments (#7268) Signed-off-by: Magiceses <magiceses0118@gmail.com> * [CHORE] adding user agent (#7281) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * receive/multitsdb: do not delete not uploaded blocks (#7166) * receive/multitsdb: do not delete not uploaded blocks If a block hasn't been uploaded yet then tell the TSDB layer not to delete them. This prevents a nasty race where the TSDB layer can delete a block before the shipper gets to it. I saw this happen with a very small block. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * receive/multitsdb: change order Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * shipper/receive: just use a single lock Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Compact: Replace group with resolution in ownsample metrics (#7283) Compaction dowsnample metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a resolution label to reduce cardinality. Fixes: #5841 Signed-off-by: SuperQ <superq@gmail.com> * Propagate warnings from instant queries Warnings from remote instant queries get turned into errors, which is a bug. It should be up to the root client to decide whether warnings should be show as such, or converted to errors. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Receive: stop relying on grpc server config to set grpc client secure/skipVerify Signed-off-by: Guillaume Lecerf <guillaume.lecerf@iziwork.com> * Show warnings in query frontend (#7289) * Show warnings in query frontend QFE currently does not parse warnings from downstream queriers. This commit fixes that by adding the field to proto messages and modifies the merge function to take warnings into account. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add CHANGELOG entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Omit empty warnings Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Optimize empty posting check in lazy posting (#7298) * change lazy postings empty posting check to use cardinality Signed-off-by: Ben Ye <benye@amazon.com> * update lazy posting test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Sidecar: mark as unqueryable if prometheus is down (#7297) If the prometheus that belongs to a sidecar is down we dont need to query the sidecar. This PR makes it so that we take the sidecar out of the endpoint set then. We do the same for all other store APIs by retuning an error in the info/Info gRPC call if they are marked as not ready. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * [CHORE] considering X-Forwarded-For on HTTP Logging (#7303) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> * Query|Receiver: Do not log full request on ProxyStore by default (#7305) * Query|Receiver|Store: Do not log full request on ProxyStore by default We had a problem on our production where a sudden increase in requests with long matchers was putting our receivers under a lot of pressure. Upon checking profiles we saw that the problem was calls to Log() S…

matej-g added good first issue help wanted component: compact labels Oct 31, 2022

SuperQ mentioned this issue Jan 17, 2023

Compact: Replace group with resolution in compact metrics. #6049

Merged

2 tasks

fpetkovski closed this as completed in #6049 Jun 17, 2023

SuperQ mentioned this issue Apr 16, 2024

Compact: Replace group with resolution in downsample metrics #7283

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compaction TODO metrics have too high a cardinality #5841

Compaction TODO metrics have too high a cardinality #5841

SuperQ commented Oct 31, 2022

yeya24 commented Oct 31, 2022

SuperQ commented Oct 31, 2022

Akshit42-hue commented Oct 31, 2022

yeya24 commented Oct 31, 2022

yeya24 commented Nov 1, 2022

SuperQ commented Nov 15, 2022

yeya24 commented Jan 17, 2023

SuperQ commented Jan 17, 2023

Compaction TODO metrics have too high a cardinality #5841

Compaction TODO metrics have too high a cardinality #5841

Comments

SuperQ commented Oct 31, 2022

yeya24 commented Oct 31, 2022

SuperQ commented Oct 31, 2022

Akshit42-hue commented Oct 31, 2022

yeya24 commented Oct 31, 2022

yeya24 commented Nov 1, 2022

SuperQ commented Nov 15, 2022

yeya24 commented Jan 17, 2023

SuperQ commented Jan 17, 2023