Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support setting the Gossip cluster "label" #3254

Closed
wants to merge 1 commit into from

Conversation

greed42
Copy link

@greed42 greed42 commented Feb 15, 2023

This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS.

It should solve issues such as #2250, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in #2250 (comment).

You can enable with the command-line flag, --cluster.gossip-label; any non-empty string will form an effective namespace for gossip communication.

If you use Prometheus Operator, you can set the ALERTMANAGER_CLUSTER_GOSSIP_LABEL environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like:

kind: Alertmanager
...
spec:
  ...
  containers:
    - name: alertmanager
      env:
        - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL
          value: infrastructure-eu-west-2
  ...

This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use.

This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS.

It should solve issues such as <prometheus#2250>, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in <prometheus#2250 (comment)>.

You can enable with the command-line flag, `--cluster.gossip-label`; any non-empty string will form an effective namespace for gossip communication.

If you use Prometheus Operator, you can set the `ALERTMANAGER_CLUSTER_GOSSIP_LABEL` environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like:

```
kind: Alertmanager
...
spec:
  ...
  containers:
    - name: alertmanager
      env:
        - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL
          value: infrastructure-eu-west-2
  ...
```

This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use.

Signed-off-by: Graham Reed <greed@7deadly.org>
@simonpasquier
Copy link
Member

Superseded by #3354

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants