Feasibility of having Thanos Receiver at scale

[varun-krishna]

Exploring ways to have Prometheus fully stateless at scale .

Present setup : Thanos , GCS for Obj Storage , 2 HA Prometheus instances .

Would like to understand how scalable receiver would be for such a setup and the tradeoff from moving away from Thanos sidecar to receiver to achieve full statelessness for Prometheus (Worst case - loss in 2 Hrs of data)

[bwplotka]

Sure (:

2 instances will work without 0 problems. We currently run Thanos Receive Cluster with ~5k Prometheus instances.. and 3x replication and it has still room to scale up (which we do every day) (:

For details on ongoing work and documentation please visit #1093 and #thanos Slack channel 🤗