Kubernetes Pods not visible under targets section

[sojinss4u]

Hello,

To obtain memory profiles from my Go application, I enabled the K8s SD scrape configuration in the parca.yaml file using the configuration below. Despite enabling the scrape configs, no pods were listed under the 'kubernetes-pods' section. However, I was still able to see some memory profiles in the UI, even though no targets appeared there.

- job_name: kubernetes-pods
    scrape_interval: "60s"
    profiling_config:
        pprof_config:
          process_cpu:
            enabled: false
        process_cpu:
          enabled: false
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_app_kubernetes_io_(.+)
        replacement: "app_kubernetes_io_$1"
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container

However I started seeing targets listed under the 'kubernetes-pods' section in UI after modifying the scrape configuration to only target pods with a label 'parca.io/scrape: "true"'. There is only one application in this cluster with this label. I suspect that the UI is not listing the targets when the number of targets it need to scrape (pod + url combination) is too large.

 - job_name: kubernetes-pods
      scrape_interval: "60s"
      profiling_config:
        pprof_config:
          process_cpu:
            enabled: false
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_parca_io_scrape__]
          regex: true
          action: keep
        - action: labelmap
          regex: __meta_kubernetes_pod_label_app_kubernetes_io_(.+)
          replacement: "app_kubernetes_io_$1"
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_container_name]
          action: replace
          target_label: container

For more details, please refer to this thread.