Add File Service Discovery

[bwplotka]

Rationales
Similarly to Prometheus File SD our idea for File SD will allow to update hosts for various purposes (StoreAPIs for Querier, Queriers for Ruler) to be discovered on runtime. Additionally it will allow to build custom integrations that will fill those in run time.

Iinitially I thought we could just use Prometheus package but it is quite tied to targets, so no longer sure. Sticking to same library will really help us to not maintain it and have proven solution that works with ANY integration that is already build into that. We should consider this.

Acceptance Criteria

• Agree if we want to reuse (and hack) same package or copy some code for something similar.
• Define if we support one file or more? Prometheus format
• Define file format. Should it be same as Prometheus one? (which will allow reusing similar sidecars/projects like this)
• Add working File Service Discovery (SD) with tests.
  • A way to dynamically specify FQDN, hosts, IPs and ports that will be resolved to IPs.
    • Support SRV
    • Support periodic resolution refresh (we can reuse Prom code: https://github.com/prometheus/prometheus/blob/master/discovery/dns/dns.go#L167)
  • Reuse or even USE runtime reload logic from Prometheus (https://github.com/prometheus/prometheus/blob/master/discovery/file/file.go#L172)
  • Hook it in:
    • Thanos Querier (for StoreAPIs)
    • Thanos Ruler (to discover Querier)
  • Document usage.
  • Add solid unit tests to confirm usability.

[bwplotka]

This is related to #484

(in fact we want to combine DNS SD as well here)

[bwplotka]

We need to also make sure this is #452 within the scope.

[yuecong]

could we also support https as prometheus's tls_config and basic_auth? With these, I think we can even config thanos cluster discovery over the internet.

[xjewer]

We need to also make sure this is #452 within the scope.

removing gossip/memberlist solves this problem, as it's all about memberlist's resolver

[bwplotka]

@xjewer Ah good point. But we will introduce another place for DNS resolution, so still it would be nice to remember about /etc/hosts (:

@yuecong I don't get this. How this is connected to File SD?

[xjewer]

so still it would be nice to remember about /etc/hosts (:

@Bplotka afaik, the golang unix resolver lookup Resolver.LookupHost respects a /etc/nsswitch.conf file and an order of lookup accordingly

[Ehekatl]

DNS based sd would be more than enough, I'm using headless service to publish thanos peer endpoint within k8s, even without k8s, manage several dns records by hand is not much of work.

It would be better if you make --cluster.peers= compatible with dns sd,
so my configuration--cluster.peers=peers.thanos.svc.cluster.local works the same after remove gossip.

[xjewer]

Is anyone who works on this issue or has plan doing it?

[ivan-valkov]

Hey, I am working on file sd right now. I am working on getting a basic version of file sd running without dns resolution. When this is done, I plan on implementing a dns resolver that can be used in the file service discovery.

[bwplotka]

This was being merged! 🎉

Thanks @ivan-valkov for this!