sidecar/receive questions

[calestyo]

Hey there.

I had asked these (and some more questions) before on slack, but there wasn't really much outcome.

When using sidecar to get the data into thanos:

1. Does sidecar need to run locally on the node where prometheus runs, or could I just mount the TSDB area from that prometheus via e.g. sshfs or nfs?
   I just wonder, because e.g. prometheus itself warns from using remote filesystems, as that would allegedly cause corruptions.
2. Why does Thanos want to restart Prometheus (i.e. why is --web.enable-lifecycle needed)?
   What happens if I wouldn’t enabled that?
3. Docs say it needs --web.enable-admin-api, but that seems to allow admin control over Prometheus, which in principle I wouldn't want.
   What happens without it?

AFAIU, I could use receive to get a true read-only mode (i.e. without the need for --web.enable-admin-api and/or --web.enable-lifecycle needed. Actually I’d plan to run one Prometheus instance using sidecar and another using receive. Both would scrape the same data and run in HA mode, with the one running sidecar being there to not loose any data in case there is some network interruption between prometheus and thanos.

At least it's my understanding that thanos would pick up the data from the the compacted TSDB buckets once the connection would be back?!

4. When I use --objstore.config-file with sidecar respectively receive and want(=need) to use FILESYSTEM a storage backend,... same as above, can I use a remote fs with that (e.g. sshfs or nfs)?

Thanks,
Chris.