ExternalLabels in MSO

[cmwylie19]

Test ExternalLabels

We test our ExternalLabels by sending metrics to a RemoteWrite endpoint and ensuring ExternalLabels defined on the MonitoringStack Operator exist in the metrics. We will deploy a custom Blue App, that has an endpoint /alerts that receives metrics. We can verify ExternalLabels are working by checking the logs of the Blue App.

• Setup Kind Cluster
• Deploy Blue Application
• Deploy MSO
• Verify ExternalLabels exist in RemoteWrite Metrics
• Clean Up

Setup Kind Cluster

To setup kind cluster, read the README.md

Deploy Blue Application

kubectl apply -f -<<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: blue
    version: v1
  name: blue
  namespace: default
spec:
  ports:
    - port: 9000
      name: http
  selector:
    app: blue
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: blue
    version: v1
  name: blue
  namespace: default
spec:
  selector:
    matchLabels:
      app: blue
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: blue
        version: v1
    spec:
      serviceAccountName: blue
      containers:
        - image: docker.io/cmwylie19/go-metrics-k8s:latest
          name: blue
          ports:
            - containerPort: 9000
              name: http
          imagePullPolicy: Always
      restartPolicy: Always
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: blue
EOF

Deploy MSO

We have defined our ExternalLabels and the RemoteWrite Endpoint.

kubectl apply -f -<<EOF
apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
  name: blue
spec:
  logLevel: debug
  prometheusConfig:
    externalLabels:
      tenant_id: "blue"
      user: "blue"
      clusterID: "blue"
    remoteWrite:
    - url: http://blue.default.svc.cluster.local:9000/alert # My app to read the metrics
  retention: 120h
EOF

Deploy ServiceMonitor

Tell Prometheus to Scrape the Blue App

kubectl apply -f -<<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blue
  labels:
    app: blue
spec:
  selector:
    matchLabels:
      app: blue
  endpoints:
  - port: http
EOF

Verify ExternalLabels exist in RemoteWrite Metrics

Since Prometheus has been instruected to send RemoteWrite metrics to the blue app, we can check the logs of the blue app to ensure that the ExternalLabels we defined are present: tenant_id, user, and clusterID

kubectl logs -l app=blue | egrep 'clusterID|tenant_id|user'

output

scrape_duration_seconds{clusterID="blue", container="blue", endpoint="http", instance="10.244.1.18:9000", job="blue", namespace="default", pod="blue-b955b9f5f-s8kjd", prometheus="default/blue", prometheus_replica="prometheus-blue-0", service="blue", tenant_id="blue", user="blue"}
scrape_samples_scraped{clusterID="blue", container="blue", endpoint="http", instance="10.244.1.18:9000", job="blue", namespace="default", pod="blue-b955b9f5f-s8kjd", prometheus="default/blue", prometheus_replica="prometheus-blue-0", service="blue", tenant_id="blue", user="blue"}
scrape_samples_post_metric_relabeling{clusterID="blue", container="blue", endpoint="http", instance="10.244.1.18:9000", job="blue", namespace="default", pod="blue-b955b9f5f-s8kjd", prometheus="default/blue", prometheus_replica="prometheus-blue-0", service="blue", tenant_id="blue", user="blue"}
scrape_series_added{clusterID="blue", container="blue", endpoint="http", instance="10.244.1.18:9000", job="blue", namespace="default", pod="blue-b955b9f5f-s8kjd", prometheus="default/blue", prometheus_replica="prometheus-blue-0", service="blue", tenant_id="blue", user="blue"}

Clean Up

kubectl delete servicemonitor blue
kubectl delete monitoringstack blue
kubectl  delete svc,deploy,sa blue
kind delete cluster --name=kind
kind delete cluster --name=kind-kind

[cmwylie19]

Closes #125

[sthaha]

@cmwylie19 Thank you for the patch and it looks good to me.

• I think we will benefit from adding an e2e test to ensure this feature works as expected.
• Could you please fix the CI issues?

For the commit lint git commit -s :

feat: support ExternalLabels prometheus config

A short description of what is changed ... 

Fixes: #125 
Signed-By: ...

Please feel free to ping us should you need help with any of this.

[cmwylie19]

@cmwylie19 Thank you for the patch and it looks good to me.

• I think we will benefit from adding an e2e test to ensure this feature works as expected.
• Could you please fix the CI issues?

For the commit lint git commit -s :

feat: support ExternalLabels prometheus config

A short description of what is changed ... 

Fixes: #125 
Signed-By: ...

Please feel free to ping us should you need help with any of this.

Absolutely, i will get to this. Thank you @sthaha

[cmwylie19]

@sthaha @jan--f would you be able to check this now? I have fixed the commits and the e2e tests. Thank you!

[sthaha]

Thanks for the patch @cmwylie19
Looks good to me!