Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUGFIX: Avoid duplicate Prometheus scrape targets by using a named port in the ServiceMonitor #471

Conversation

wallrj
Copy link
Member

@wallrj wallrj commented Jul 25, 2024

In #229 @leotomas837 reported that:

The ServiceMonitor targets both the web hook service and the metrics service. Yet, only the metrics service must be scraped (the probe could be scraped too via the blackbox-exporter, but that is a subject for another PR). The webhook service must not be discovered by Prometheus.

In that original PR, we tried to fix this by modifying the labels in the Service and ServiceMonitor, but while testing it we found a simpler solution, which is to specify a port name in the ServiceMonitor.

Before
Screenshot 2024-07-25 094542

After
Screenshot 2024-07-25 100019

Testing

You can test this yourself as follows:

# Start a Kind cluster and install approver-policy (from source) + cert-manager
make test-smoke-deps

# Deploy  Prometheus
helm upgrade default kube-prometheus-stack \
    --repo https://prometheus-community.github.io/helm-charts \
    --install \
    --namespace prometheus \
    --create-namespace \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --wait

# Redeploy approver-policy with ServiceMonitor enabled
make test-smoke-deps \
  INSTALL_OPTIONS="--set image.repository=\$(oci_manager_image_name_development) --set app.metrics.service.servicemonitor.enabled=true"

# Connect to prometheus web UI
kubectl port-forward -n prometheus pods/prometheus-default-kube-prometheus-st-prometheus-0 9090

Visit: http://localhost:9090/targets

…scrape targets

Signed-off-by: Richard Wall <richard.wall@venafi.com>
@cert-manager-prow cert-manager-prow bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 25, 2024
@wallrj wallrj changed the title BUGFIX: Prevent duplicate Prometheus scrape targets by using a named port in the ServiceMonitor BUGFIX: Avoid duplicate Prometheus scrape targets by using a named port in the ServiceMonitor Jul 25, 2024
@wallrj wallrj requested a review from SgtCoDFish July 25, 2024 11:08
Copy link
Member

@SgtCoDFish SgtCoDFish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

I didn't go into any great detail with installing this, but it looks like what I'd expect from looking at the docs! Cheers!

@cert-manager-prow cert-manager-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jul 25, 2024
@cert-manager-prow
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SgtCoDFish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 25, 2024
@cert-manager-prow cert-manager-prow bot merged commit 6763400 into cert-manager:main Jul 25, 2024
5 checks passed
@wallrj wallrj deleted the fix-duplicate-prometheus-scrape-targets branch July 25, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants