Missing controlplane metrics: opentelemetry-kube-stack-0.2.1

[suresh-target]

It is observed that the metrics of controlplane components such as apiserver, etcd are missing when we use the TargetAllocator with the help of ServiceMonitor CRDs

helm chart - opentelemetry-kube-stack-0.2.1

values.yaml configuration

kubernetesServiceMonitors:
  enabled: true
kubeApiServer:
  enabled: true
kubeEtcd:
  enabled: true

Looks like we may need a separate TargetAllocator instance to scrape controlplane metrics as the per-node strategy currently ignores targets without a Node, like control plane components.

Notes:

1. serviceMonitor.additionalLabels also need to be enabled to add labels to ServiceMonitor instances so that TA instance can use them to filter a specific set of Servicemonitors.
2. This was tested by adding a missing label manually to both apiserver and etcd services to overcome the issue mentioned here

Additional observations

The following curl command can be executed to confirm that it returns empty target list

# curl http://opentelemetry-kube-stack-daemon-targetallocator.kube-system.svc.cluster.local/jobs/serviceMonitor%2Fkube-system%2Fopentelemetry-kube-stack-kube-etcd%2F0/targets

{"opentelemetry-kube-stack-daemon-collector-dpst2":{"_link":"/jobs/serviceMonitor%2Fkube-system%2Fopentelemetry-kube-stack-kube-etcd%2F0/targets?collector_id=opentelemetry-kube-stack-daemon-collector-dpst2","targets":[]},"opentelemetry-kube-stack-daemon-collector-d7qsq":{"_link":"/jobs/serviceMonitor%2Fkube-system%2Fopentelemetry-kube-stack-kube-etcd%2F0/targets?collector_id=opentelemetry-kube-stack-daemon-collector-d7qsq","targets":[]},"opentelemetry-kube-stack-daemon-collector-pm5qq":{"_link":"/jobs/serviceMonitor%2Fkube-system%2Fopentelemetry-kube-stack-kube-etcd%2F0/targets?collector_id=opentelemetry-kube-stack-daemon-collector-pm5qq","targets":[]}}

[jaronoff97]

@suresh-target i think this issue is separate from #1371 but want to confirm. You are indeed correct that you will need to run a separate TA instance if your strategy is per-node, I don't think that's something we would support as part of this chart, however.

[suresh-target]

@jaronoff97 Yes, it is separate from #1371.

[suresh-target]

@jaronoff97 Update: I have fixed this problem by adding one more collector that has a TA instance which is running in statefulset mode. Now I noticed that the field serviceMonitor.additionalLabels is not enabled, enabling this label may help us to add custom labels to serviceMonitor instance so that we can segregate them as needed.

The plan is to enable TA to use a label to lookup a specific group of serviceMonitor instances, for example

Add a custom label to serviceMonitor

serviceMonitor:
    additionalLabels:
      component: control-plane

Then use that label in TA configuration to lookup for a serviceMonitor that has this label

serviceMonitorSelector:
          matchLabels:
             component: control-plane

[suresh-target]

Submitted a PR to add additional label #1382

[jaronoff97]

@suresh-target PR is merged, i think we should be able to close this. Please let me know if you have more issues! thank you for your contribution 🙇