Clustering: re-discover and re-join peers on interval in background

[rfratto]

Request

Add a new flag, --cluster.rejoin-interval which specifies how often a node should rediscover peers and rejoin them to address split brain issues.

--cluster.rejoin-interval should default to some reasonable value, such as 60s.

When set to 0s, rediscovery/rejoining is disabled.

This proposal should be paired with #4464 to avoid overwhelming the network on large clusters, as the state push/pull done on join is more expensive than other gossip traffic.

Use case

Today, for nodes to join a cluster successfully, one of the following must be true:

• All nodes must join the same seed node, or
• The initial cluster cannot be bootstrapped in parallel.

This is because the set of peers to join is determined once at startup.

In effect, this means that clustering on Kubernetes must only be enabled via a StatefulSet, and podManagementPolicy must not be set to Parallel. If neither of these conditions are met, clustering will end up in a split brain state.

To avoid these constraints, nodes should rediscover and rejoin the cluster in the background on some timer to address split brain issues.

[braunsonm]

Along with this issue, please add support to the chart for the additional flags along with the clustering flag that was recently added

[tpaschalis]

@braunsonm are you suggesting exposing the rest of the --cluster.* flags along the clustering.enabled field in values.yaml? We've thought of expanding it for ease of use once we've settled on the options we'd like to expose.

In the meantime, there's an extraArgs field on values.yaml you can use to pass any flags to the run command.

[braunsonm]

Yea I'm suggesting clustering.enabled in the chart should configure auto discovery on an interval in the future.

Can definitely be done manually in the meantime, but ideally this should allow more modes than just statefulset to be deployed. We're specifically looking for daemonset to be supported as we run node exporter with Grafana Agent