Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[alertmanager] Add prometheus servicemonitor support #4631

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Arabus
Copy link
Contributor

@Arabus Arabus commented Jun 26, 2024

What this PR does / why we need it

Add servicemonitor support to alertmanager

Which issue this PR fixes

Special notes for your reviewer

Checklist

  • DCO signed
  • Chart Version bumped
  • Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

Signed-off-by: Bernd May <b.may@syseleven.de>
@Arabus
Copy link
Contributor Author

Arabus commented Jun 26, 2024

@monotek it appears the lint check failed for no apparent reason?

@Arabus
Copy link
Contributor Author

Arabus commented Jun 27, 2024

nvm i see the errors now; though they are not related to anything I touched and I don't see what the linter wants me to do. there appear to be some problems wit the comments around l 115 and 116 🤷

Signed-off-by: Bernd May <b.may@syseleven.de>
@Arabus
Copy link
Contributor Author

Arabus commented Jun 27, 2024

mucking around with yamllint did the trick but now I face the consequence of adding a resource that requires the prometheus crd for a service monitor... so either I add the crd chart s as a dependency (preferred) or exclude the resource based on a capabilities test.

WDYT?

We cannot assume that prometheus is installed therefore the user
needs to decide this consciously

Signed-off-by: Bernd May <b.may@syseleven.de>
Signed-off-by: Bernd May <b.may@syseleven.de>
@Arabus Arabus force-pushed the syseleven/add_smon_to_alertmanager branch from 9a22ddd to f9a7399 Compare June 27, 2024 07:20
@Arabus
Copy link
Contributor Author

Arabus commented Jun 28, 2024

well should work now - please have a look

@DrFaust92
Copy link
Contributor

my 2 cents,
I think the AM chart shouldnt bundle crds, same as prom chart (non operator) doesnt assume CRDs or tries to install them. it should "assume" their existence and install svc monitor if api is available, like so

{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) ( .Values.serviceMonitor.enabled ) }}
this way its loosely coupled

Copy link
Member

@monotek monotek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see code comments...

charts/alertmanager/Chart.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
charts/alertmanager/values.yaml Outdated Show resolved Hide resolved
@Arabus
Copy link
Contributor Author

Arabus commented Jul 4, 2024

my 2 cents, I think the AM chart shouldnt bundle crds, same as prom chart (non operator) doesnt assume CRDs or tries to install them. it should "assume" their existence and install svc monitor if api is available, like so

{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) ( .Values.serviceMonitor.enabled ) }}

this way its loosely coupled

That was the other alternative I considered; I'm fine with either way. This contains the 'risk' of not deploying the servicemonitor even though someone enabled it though. I have added an additional if - fail instead; that way one has to have the capability or explicitly fake it with --api-versions

In order to automatically monitor alertmanager with prometheus and
prometheus operator we need a servicemonitor resource.

Signed-off-by: Bernd May <b.may@syseleven.de>
@Arabus Arabus force-pushed the syseleven/add_smon_to_alertmanager branch from ef820ef to 8b49899 Compare July 4, 2024 08:02
monotek
monotek previously approved these changes Jul 5, 2024
@Arabus
Copy link
Contributor Author

Arabus commented Jul 9, 2024

I noticed a side effect of the current change; due to the chosen label selectors we get duplicate metrics. The headless and regular services have the same labels, which is why the servicemonitors label selector matches both.

In order to distinguish them, I could add a label to the headless service. I could also switch to using a pod monitor.

There is also the option of using a targetLimit though I have never used that and will need to test if it does what we want.

Signed-off-by: Bernd May <b.may@syseleven.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[alertmanager] Provide servicemonitor template
3 participants