Promxy not working with Kube prometheus stack

[MChoudhary478062]

While trying to upgrade prometheus-operator to kube-prometheus-stack it was observed that promxy is not working as intended where is is not discovering the prometheus pods.

Here is some additional information:

helm version

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

kubectl -n monitoring get pods [Output limited to promxy and kube-prometheus-stack]

NAME READY STATUS RESTARTS AGE
prom-op-promxy-66b6fb47d8-l7fxt 2/2 Running 0 8h
prometheus-prom-op-kube-prometheus-st-prometheus-0 2/2 Running 0 8h
prometheus-prom-op-kube-prometheus-st-prometheus-1 2/2 Running 0 8h

]# kubectl -n monitoring get sa
NAME SECRETS AGE
promxysa 1 8h

kubectl -n monitoring get deploy

NAME READY UP-TO-DATE AVAILABLE AGE
prom-op-promxy 1/1 1 1 8h

kubectl -n monitoring get clusterRole | grep -i "promxy"

prom-op-promxy-cr 2022-06-29T16:42:35Z

kubectl -n monitoring get clusterRoleBinding | grep -i "promxy"

prom-op-promxy-crb ClusterRole/prom-op-promxy-cr 8h

kubectl -n monitoring get cm/prom-op-promxy -o yaml

apiVersion: v1
data:
config.yaml: |-
global:
evaluation_interval: 5s
external_labels:
source: prom-service
promxy:
server_groups:
- kubernetes_sd_configs:
- namespaces:
names:
- monitoring
role: pod
relabel_configs:
- action: keep
regex: (prometheus)
replacement: $1
separator: ;
source_labels:
- __meta_kubernetes_pod_label_app
remote_write:
- queue_config:
batch_send_deadline: 500ms
capacity: 500000
max_backoff: 2000ms
max_samples_per_send: 10000
max_shards: 1
min_backoff: 100ms
remote_timeout: 30s
url: http://XXXX:8080/receive
write_relabel_configs:
- action: keep
regex: XXXX
source_labels:
- name
rule_files:
- /etc/promxy-rules/*.rules
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: prom-op
meta.helm.sh/release-namespace: monitoring
creationTimestamp: "2022-06-29T16:42:34Z"
labels:
app.kubernetes.io/instance: prom-op
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: promxy
app.kubernetes.io/version: 0.0.75
helm.sh/chart: promxy-1.0.0
name: prom-op-promxy
namespace: monitoring
resourceVersion: "4302519"
uid: ae45ae3e-fd62-4e98-887d-640559950f41

Seems that kubernetes_sd_configs is not able to discover the kube-prometheus pods.
Note: Both promxy & kube-prometheus pods are running in the monitoring namespace

[jacksontj]

I was unable to reproduce this issue locally in #568

My guess is that this is an issue with your role or role binding -- but we'll need more information to debug.

Given that this is a fairly old issue I'll close this out for now, but if you are still having this issue please re-open!