Store,Compactor: Thanos sharding #1583

jojohappy · 2019-09-30T04:01:22Z

CHANGELOG entry if change is relevant to the end user.

Changes

Implement partial parts of proposal #1501 :

Add relabel config into Thanos, store gateway and compactor are supported now.
Selecting blocks to serve depends on the result of block labels relabeling.
For store gateway, expose advertise labels after relabeling.
Allow the duplicate external labels for store gateway components.

Note: pre-filter has been implemented before this PR.

TODO:

Add unit test.
Add documents for relabeling.

@bwplotka @brancz @povilasv @GiedriusS PTAL, thanks!

bwplotka · 2019-10-03T09:43:27Z

Nice! It looks awesome generally. Will try to look on it ASAP.

bwplotka

Nice, generally it looks like a good start! Some suggestions (: Thanks for doing this!

bwplotka · 2019-10-06T10:35:00Z

pkg/compact/compact.go

@@ -275,6 +280,15 @@ func (c *Syncer) downloadMeta(ctx context.Context, id ulid.ULID) (*metadata.Meta
 		return nil, errors.Wrapf(err, "downloading meta.json for %s", id)
 	}

+	// Check for block labels by relabeling.


Nice. However I would move it out of downloadMeta as we do more than method suggests which might be confusing. Let's do it after meta is downloaded? Also it's nice to do it after downloadMeta method as we check if it's not too fresh first.

bwplotka · 2019-10-06T10:36:46Z

pkg/query/storeset.go

+	// Append the storeType to the end of list, because the store gateway will be exposed external labels,
+	// we allow the duplicate external labels with different components.
+	if store.storeType != nil {
+		tsdbLabelSetStrings = append(tsdbLabelSetStrings, store.storeType.String())


Hm, this sounds like a hack, there is no way to check if this guy is a storeGW inside checking deduplicated stuff?

Sorry, what do you mean?

We are adding some artificial labels to the labelset. Why? This sounds wrong? To allow duplicated store gateway labels we just need to avoid detecting this if store.storeType is Gateway, right? NOT

bwplotka · 2019-10-06T10:40:58Z

pkg/store/bucket.go

+		level.Debug(s.logger).Log("msg", "dropping block(drop in relabeling)", "block", id)
+		return os.RemoveAll(dir)
+	}
+	b.labels = labels.FromMap(processedLabels.Map())


This is important because actually we get the labels after relabelling, which we agreed not doing in the first iteration (:

So should we expose the original labels from meta.json? Maybe the action of relabeling will be labeldrop or labelkeep.

Yes!

It may yes. But what's the use case to drop only some labels?

In fact it would be more useful to add new labels! This is great but there are some limitiation for it (e.g relabelling cannot generate 2 lablesets) . For now in this PR I would say we care ONLY if it exists or not exists. Does it sounds fair? (:

bwplotka · 2019-10-06T10:41:19Z

pkg/store/bucket.go

-	}, nil
+	}
+
+	if len(s.relabelConfig) != 0 {


I would vote to precompute this every sync instead of computing this on every Info which is frequent (at least every 5s)

… gateway Signed-off-by: jojohappy <sarahdj0917@gmail.com>

…mpty labels when no relabel configuration Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

bwplotka · 2019-10-08T17:16:59Z

It would really nice to add tests particulary for sharding compactor and store GW. Maybe in /test/e2e/?

bwplotka

Awesome! It looks amazing. Hopefully last things to fix!

Thanks for doing it!

bwplotka · 2019-10-08T21:58:19Z

CHANGELOG.md

@@ -20,6 +20,10 @@ We use *breaking* word for marking changes that are not backward compatible (rel
 - [#1534](https://github.com/thanos-io/thanos/pull/1534) Thanos Query Added `/-/ready` and `/-/healthy` endpoints.
 - [#1533](https://github.com/thanos-io/thanos/pull/1533) Thanos inspect now supports the timeout flag.
 - [#1362](https://github.com/thanos-io/thanos/pull/1362) Optional `replicaLabels` param for `/query` and `/query_range` querier endpoints. When provided overwrite the `query.replica-label` cli flags.
+- [#1583](https://github.com/thanos-io/thanos/pull/1583) Thanos sharding:
+  - Add relabel config (`--selector.relabel-config-file` and `selector.relabel-config`) into Thanos


Worth to note exact components (:

bwplotka · 2019-10-08T21:58:47Z

CHANGELOG.md

+- [#1583](https://github.com/thanos-io/thanos/pull/1583) Thanos sharding:
+  - Add relabel config (`--selector.relabel-config-file` and `selector.relabel-config`) into Thanos
+  - Selecting blocks to serve depends on the result of block labels relabeling.
+  - For store gateway, expose advertise labels after relabeling.


Suggested change

- For store gateway, expose advertise labels after relabeling.

- For store gateway, advertise labels from "approved" blocks.

bwplotka · 2019-10-08T21:59:08Z

CHANGELOG.md

@@ -20,6 +20,10 @@ We use *breaking* word for marking changes that are not backward compatible (rel
 - [#1534](https://github.com/thanos-io/thanos/pull/1534) Thanos Query Added `/-/ready` and `/-/healthy` endpoints.
 - [#1533](https://github.com/thanos-io/thanos/pull/1533) Thanos inspect now supports the timeout flag.
 - [#1362](https://github.com/thanos-io/thanos/pull/1362) Optional `replicaLabels` param for `/query` and `/query_range` querier endpoints. When provided overwrite the `query.replica-label` cli flags.
+- [#1583](https://github.com/thanos-io/thanos/pull/1583) Thanos sharding:
+  - Add relabel config (`--selector.relabel-config-file` and `selector.relabel-config`) into Thanos
+  - Selecting blocks to serve depends on the result of block labels relabeling.


This should be part of the last sentence.

bwplotka · 2019-10-08T22:00:06Z

cmd/thanos/flags.go

+	return extflag.RegisterPathOrContent(
+		cmd,
+		"selector.relabel-config",
+		"YAML file that contains seletor relabeling configuration. It follows native Prometheus relabel-config syntax. See format details: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config ",


Suggested change

"YAML file that contains seletor relabeling configuration. It follows native Prometheus relabel-config syntax. See format details: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config ",

"YAML file that contains relabeling configuration that allows selecting blocks. It follows native Prometheus relabel-config syntax. See format details: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config ",

bwplotka · 2019-10-08T22:08:26Z

pkg/query/storeset.go

@@ -218,7 +218,10 @@ func (s *StoreSet) Update(ctx context.Context) {
 	// Record the number of occurrences of external label combinations for current store slice.
 	externalLabelOccurrencesInStores := map[string]int{}
 	for _, st := range healthyStores {
-		externalLabelOccurrencesInStores[externalLabelsFromStore(st)]++
+		if st.storeType != nil && (st.storeType.ToProto() == storepb.StoreType_QUERY ||


Why are we changing this? Let's maybe focus on line 258 as you did instead (:

Thanks! Done!

bwplotka · 2019-10-08T22:09:50Z

pkg/query/storeset.go

-		if len(store.LabelSets()) > 0 && externalLabelOccurrencesInStores[externalLabels] != 1 {
+		if len(store.LabelSets()) > 0 &&
+			store.storeType != nil &&
+			(store.storeType.ToProto() == storepb.StoreType_QUERY ||


Can we just exclude StoreType_STORE? It will be much safe. It's the only thing that we change here - store GW handing, right?

bwplotka · 2019-10-08T22:11:11Z

pkg/store/bucket.go

-	}, nil
+	}
+
+	labelSets := make(map[uint64][]storepb.Label, len(s.blocks))


Again as I mentioned before - we can build that every sync instead of every Info call. Can we do that? (:

Info should be as fast as possible as it works as healthiness/heartbeat check.

I will update today. Thanks!

bwplotka · 2019-10-08T22:15:34Z

It would be also awesome to have some tests for compactor filtering and store GW exposing info + filtering. Maybe in next PR (:

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

bwplotka · 2019-10-09T13:38:12Z

Tests fails ):

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

jojohappy · 2019-10-09T15:42:05Z

@bwplotka That's green!

bwplotka

Awesome! Good work, LGTM

* Add selector.relabel-config flag && expose advertised label for store gateway Signed-off-by: jojohappy <sarahdj0917@gmail.com> Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

xiaozongyang · 2020-12-22T10:11:03Z

Hi @jojohappy @bwplotka
Do we have some examples of selecting blocks with relabel configs here?

jojohappy requested review from bwplotka, povilasv, brancz and GiedriusS September 30, 2019 04:03

bwplotka approved these changes Oct 6, 2019

View reviewed changes

jojohappy added 8 commits October 8, 2019 22:08

Add selector.relabel-config flag && expose advertised label for store…

f99502e

… gateway Signed-off-by: jojohappy <sarahdj0917@gmail.com>

To allow the duplicate external labels with different components && E…

94d02ca

…mpty labels when no relabel configuration Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Remove dropped block dir

44483d6

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Compact: support sharding compactor

65da803

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Add CHANGELOG

6a95f51

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Fixed docs and lints

b560510

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Fixed when no specific storeType

f0ce02a

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Add test

b52a238

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

jojohappy changed the title ~~[WIP]: Store,Compactor: Thanos sharding~~ Store,Compactor: Thanos sharding Oct 8, 2019

Rebase master && update storeset test

a210289

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

jojohappy force-pushed the store_relabeling branch from d420725 to a210289 Compare October 8, 2019 16:01

jojohappy added 2 commits October 9, 2019 00:27

Update docs

76b9981

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

Fixed unstable test

d27bb87

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

bwplotka reviewed Oct 8, 2019

View reviewed changes

Address review comments

de9a47f

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

jojohappy force-pushed the store_relabeling branch from 295b12d to de9a47f Compare October 9, 2019 06:38

Add e2e test

148828d

Signed-off-by: jojohappy <sarahdj0917@gmail.com>

jojohappy force-pushed the store_relabeling branch from ecf1456 to 148828d Compare October 9, 2019 15:24

bwplotka approved these changes Oct 9, 2019

View reviewed changes

bwplotka merged commit e954fe7 into thanos-io:master Oct 9, 2019

FUSAKLA mentioned this pull request Jan 21, 2020

store: fails to load compacted block #2022

Closed

daixiang0 mentioned this pull request Jan 28, 2020

docs: Document Thanos Sharding #1922

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store,Compactor: Thanos sharding #1583

Store,Compactor: Thanos sharding #1583

jojohappy commented Sep 30, 2019 •

edited

Loading

bwplotka commented Oct 3, 2019

bwplotka left a comment

bwplotka Oct 6, 2019

bwplotka Oct 6, 2019

jojohappy Oct 6, 2019

bwplotka Oct 8, 2019

bwplotka Oct 6, 2019

jojohappy Oct 6, 2019 •

edited

Loading

bwplotka Oct 8, 2019

jojohappy Oct 8, 2019

bwplotka Oct 6, 2019

bwplotka commented Oct 8, 2019

bwplotka left a comment

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

jojohappy Oct 9, 2019

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

bwplotka Oct 8, 2019

jojohappy Oct 9, 2019

bwplotka commented Oct 8, 2019

bwplotka commented Oct 9, 2019

jojohappy commented Oct 9, 2019

bwplotka left a comment

xiaozongyang commented Dec 22, 2020

	- For store gateway, expose advertise labels after relabeling.
	- For store gateway, advertise labels from "approved" blocks.

	"YAML file that contains seletor relabeling configuration. It follows native Prometheus relabel-config syntax. See format details: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config ",
	"YAML file that contains relabeling configuration that allows selecting blocks. It follows native Prometheus relabel-config syntax. See format details: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config ",

Store,Compactor: Thanos sharding #1583

Store,Compactor: Thanos sharding #1583

Conversation

jojohappy commented Sep 30, 2019 • edited Loading

Changes

bwplotka commented Oct 3, 2019

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jojohappy Oct 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka commented Oct 8, 2019

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka commented Oct 8, 2019

bwplotka commented Oct 9, 2019

jojohappy commented Oct 9, 2019

bwplotka left a comment

Choose a reason for hiding this comment

xiaozongyang commented Dec 22, 2020

jojohappy commented Sep 30, 2019 •

edited

Loading

jojohappy Oct 6, 2019 •

edited

Loading