Enabling K8s cluster receiver

[atsikham]

Hi Team,

Could anyone help me with the following issue? I’m installing splunk-otel-collector on EKS via Helm chart and trying to configure additional K8s metrics, such as kubernetes.stateful_set.updated. For that, there is an additional receivers configuration like it’s in the doc - https://docs.splunk.com/Observability/gdi/opentelemetry/components/kubernetes-cluster-receiver.html#configuration, but not all the metrics are still visible via metrics finder.

Chart values:

clusterName: ${cluster_name}
splunkObservability:
  realm: ${realm}
  accessToken: ${signalfx_token}
cloudProvider: aws
distribution: eks
environment: ${environment}
agent:
  config:
    receivers:
      kubeletstats:
        metric_groups:
        - container
        - pod
        - node
        - volume
      k8s_cluster:
        auth_type: serviceAccount
        collection_interval: 30s
        node_conditions_to_report: ["Ready", "MemoryPressure"]
        allocatable_types_to_report: ["cpu","memory"]
        metadata_exporters: [signalfx]
      receiver_creator:
        receivers:
          prometheus_simple/plaintext:
            config:
              endpoint: '`endpoint`:`"prometheus.io/port" in annotations ? annotations["prometheus.io/port"] : 9090`'
              metrics_path: /metrics
            rule: type == "pod" && annotations["prometheus.io/export-to-signalfx"] == "true"
          smartagent/coredns:
            config:
              port: 9153
              type: coredns
            rule: type == "pod" && labels["k8s-app"] == "kube-dns"
    service:
      telemetry:
        logs:
          level: warn
  maxUnavailable: 20%
  resources:
   limits:
     memory: ${agent_memory_limit}
   requests:
     cpu: 40m
     memory: 200Mi
clusterReceiver:
  resources:
   limits:
     memory: ${receiver_memory_limit}
   requests:
     memory: ${receiver_memory_request}
serviceAccount:
  create: true
  name: ${service_account_name}

[govardha]

Hello @atsikham

I just got this working with all the metrics you are referring to. The issue is you are introducing this receiver k8s_cluster in the agent.config, it should be in clusterReceiver.config. This module listens to the kube-api server to produce the metrics, so you don't want this in your agents, which may end up duplicating the same metrics pointlessly. Hence you need to stand this up in the cluster receiver. Hope this helps.

Also the metrics shows up as

k8s.statefulset.* rather than kubernetes.statefulset.* as you expect it to be.

g

[atsikham]

Hello @govardha,

Actually, while your point is correct, there were no statefulset/job/hpa metrics at all. The reason is that they are disabled by default - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/afe04e45d4fa993fa52f862cb86d70399805a5cc/exporter/signalfxexporter/internal/translation/default_metrics.go#L99