Deduplication labels per tenant

[brancz]

Is your proposal related to a problem?

Different tenants may choose to label their ingesting Prometheus servers with different labels, Thanos should be flexible enough to allow for this.

Describe the solution you'd like

I'm inclined to allow it per request in order to be able to have different label schemes even within one tenant, on the other hand the same per-tenant deduplication-label would be good to have at offline deduplication time, so a per-tenant configuration could be a good idea as well.

Describe alternatives you've considered

Enforcing tenants to use the same labels for deduplication everywhere. This is not practical I believe as you can't necessarily enforce things like this, and even if it's likely that in the lifetime of running Thanos these might want to change.

@thanos-io/thanos-maintainers

[bwplotka]

Makes sense, thanks for raising. I think this screams for config object instead of flag though 🤔

[brancz]

Did I write flag anywhere?

[LeviHarrison]

@brancz I might take a swing at this if that's OK. I just have a few questions about the implementation:

• What I've gathered is that the proposal is to allow different deduplication labels per tenant so that when making a query encompassing multiple tenants, everything would be deduplicated properly. Is that correct?

• So there would be a configuration that looks something like this:

replica_labels:
  - tenant1:
      - replicay
  - tenant2:
      - replica1
      - replicax

And that would be in a file, specified by a flag like --query.replica-label-file=replica_labels.yaml.

• You also mentioned having an option to do this per-request, which I think might get a little complicated regarding URL parameters, but maybe bending the existing replicaLabels parameter with something along the lines of this:

?replicaLabels=tenant1=replicay&replicaLabels=tenant2=replica1

and then use a regex.

• I'm a little bit new to Thanos, how would you distinguish between tenants in the querier?

[brancz]

I think I would start with the config and not per request (it could still be done additionally later). I suspect that there will be various per tenant configurations eventually, so I think I would invert the config:

tenants:
- name: tenant1
  replica_labels:
  - replicay
- name: tenant2
  replica_labels:
  - replica1
  - replicax

We would need something like what is depicted in #3822 to identify the tenant of a request (which does not exist yet).

[LeviHarrison]

Ok, so we use the value of the THANOS_TENANT header (or something like that) in a tenant label in the query and then deduplicate the results based on the replica labels specified in the config (if there are any specified). Does that seem right?

Sidenote: is there a specific label used to distinguish tenants, or will that have to be configured also?

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.