K8s Helm deploy seems to be buggy / broke

[Ron-Norman]

Bug Report

When deployed as helm chart, most grafana dashboard panels are showing "No Data" and when I expose Prometheus and view target help, most are broken. I have not done any customizations other that creating ingress for frontend and Prometheus. If I deploy on docker it all seems to work.

Which version of the demo you are using?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
otel-demo default 2 2024-12-06 11:39:37.81596989 -0700 MST deployed opentelemetry-demo-0.33.7 1.12.0

Symptom

A clear and concise description of what the bug is.

When deployed as helm chart, most grafana dashboard panels are showing "No Data" and when I expose Prometheus and view target help, most are broken. I have not done any customizations other that creating ingress for frontend and Prometheus. If I deploy on docker it all seems to work. Also the dashboard service selector is not populated.

What do you expect to see?

Working the same as when deploy via docker.

Please describe the actual behavior experienced.

Same as above, see attach.

Could you provide the minimum required steps to resolve the issue you're seeing?

1. deploy a default k3s - I trying on a multi-node set and single simple config.
2. Deply otel demo with minimal customizations to expose the front-end and Prometheus.
3. Access Grafana and Prometheus to see issue

We will close this issue if:

• The steps you provided are complex.
• If we can not reproduce the behavior you're reporting.

Additional Context

Please feel free to add any other context about the problem here.

Here is my custom values file

components:
  frontendProxy:
    ingress:
      enabled: true
      ## Added below to fix issue with not accessible via IC
      ingressClassName: traefik
      annotations: {}
      hosts:
        - host: otel-demo-lcl.home.lab
          paths:
            - path: /
              pathType: Prefix
              port: 8080
    service:
      type: LoadBalancer
    envOverrides:
      - name: PUBLIC_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: http://otel-demo.home.lab/otlp-http/v1/traces

prometheus:
server:
service:
type: LoadBalancer
loadBalancerIP: "10.10.230.111"

Kubectl output

ronanorm@rnws-linK|opentelemetry-demo$>> kubectl get all
NAME                                                   READY   STATUS    RESTARTS        AGE
pod/otel-demo-accountingservice-996db44fd-mdbnm        1/1     Running   0               23m
pod/otel-demo-adservice-6d9788b57d-kdj44               1/1     Running   0               23m
pod/otel-demo-cartservice-7cff65c78f-fcxkb             1/1     Running   0               23m
pod/otel-demo-checkoutservice-799bfd4698-2fkbp         1/1     Running   0               23m
pod/otel-demo-currencyservice-6b48b4d447-mvh2x         1/1     Running   0               23m
pod/otel-demo-emailservice-9c6cdbdd8-xjjg2             1/1     Running   0               23m
pod/otel-demo-flagd-dd64896b-zzjmv                     2/2     Running   0               23m
pod/otel-demo-frauddetectionservice-797598cc59-2dcvc   1/1     Running   0               23m
pod/otel-demo-frontend-f9d55f999-gj27b                 1/1     Running   1 (7m15s ago)   23m
pod/otel-demo-frontendproxy-587fd84d44-z9rnb           1/1     Running   0               23m
pod/otel-demo-grafana-75f688869c-4n45d                 1/1     Running   0               23m
pod/otel-demo-imageprovider-cb8c66b4c-ttwzh            1/1     Running   0               23m
pod/otel-demo-jaeger-5f44f99657-cbm29                  1/1     Running   0               23m
pod/otel-demo-kafka-6c4cfcb854-gjnl2                   1/1     Running   0               23m
pod/otel-demo-loadgenerator-5944bf844c-6mmpz           1/1     Running   0               23m
pod/otel-demo-opensearch-0                             1/1     Running   0               23m
pod/otel-demo-otelcol-7596b9ffd7-kh9tz                 1/1     Running   0               23m
pod/otel-demo-paymentservice-7b884d56dd-tklht          1/1     Running   0               23m
pod/otel-demo-productcatalogservice-5959cd684c-7644j   1/1     Running   0               23m
pod/otel-demo-prometheus-server-78869b8654-k2fv2       1/1     Running   0               23m
pod/otel-demo-quoteservice-bfb7cb5c6-j9j69             1/1     Running   0               23m
pod/otel-demo-recommendationservice-67f8fff94c-r2tfp   1/1     Running   0               23m
pod/otel-demo-shippingservice-75c98bd59f-4qzx2         1/1     Running   0               23m
pod/otel-demo-valkey-668fdfcffc-lpmch                  1/1     Running   0               23m

NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                     AGE
service/kubernetes                        ClusterIP      10.43.0.1       <none>        443/TCP                                                                     118m
service/otel-demo-adservice               ClusterIP      10.43.131.59    <none>        8080/TCP                                                                    24m
service/otel-demo-cartservice             ClusterIP      10.43.8.176     <none>        8080/TCP                                                                    24m
service/otel-demo-checkoutservice         ClusterIP      10.43.193.209   <none>        8080/TCP                                                                    24m
service/otel-demo-currencyservice         ClusterIP      10.43.118.119   <none>        8080/TCP                                                                    24m
service/otel-demo-emailservice            ClusterIP      10.43.130.50    <none>        8080/TCP                                                                    24m
service/otel-demo-flagd                   ClusterIP      10.43.239.96    <none>        8013/TCP,4000/TCP                                                           24m
service/otel-demo-frontend                ClusterIP      10.43.36.241    <none>        8080/TCP                                                                    24m
service/otel-demo-frontendproxy           LoadBalancer   10.43.73.122    10.10.10.25   8080:32699/TCP                                                              24m
service/otel-demo-grafana                 ClusterIP      10.43.71.136    <none>        80/TCP                                                                      24m
service/otel-demo-imageprovider           ClusterIP      10.43.58.74     <none>        8081/TCP                                                                    24m
service/otel-demo-jaeger-agent            ClusterIP      None            <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP                                         24m
service/otel-demo-jaeger-collector        ClusterIP      None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP                    24m
service/otel-demo-jaeger-query            ClusterIP      None            <none>        16686/TCP,16685/TCP                                                         24m
service/otel-demo-kafka                   ClusterIP      10.43.113.132   <none>        9092/TCP,9093/TCP                                                           24m
service/otel-demo-loadgenerator           ClusterIP      10.43.221.13    <none>        8089/TCP                                                                    24m
service/otel-demo-opensearch              ClusterIP      10.43.220.1     <none>        9200/TCP,9300/TCP,9600/TCP                                                  24m
service/otel-demo-opensearch-headless     ClusterIP      None            <none>        9200/TCP,9300/TCP,9600/TCP                                                  24m
service/otel-demo-otelcol                 ClusterIP      10.43.206.95    <none>        6831/UDP,14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP,9464/TCP,9411/TCP   24m
service/otel-demo-paymentservice          ClusterIP      10.43.51.91     <none>        8080/TCP                                                                    24m
service/otel-demo-productcatalogservice   ClusterIP      10.43.31.239    <none>        8080/TCP                                                                    24m
service/otel-demo-prometheus-server       LoadBalancer   10.43.24.117    10.10.10.25   9090:32129/TCP                                                              24m
service/otel-demo-quoteservice            ClusterIP      10.43.234.227   <none>        8080/TCP                                                                    24m
service/otel-demo-recommendationservice   ClusterIP      10.43.4.64      <none>        8080/TCP                                                                    24m
service/otel-demo-shippingservice         ClusterIP      10.43.155.248   <none>        8080/TCP                                                                    24m
service/otel-demo-valkey                  ClusterIP      10.43.218.62    <none>        6379/TCP                                                                    24m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/otel-demo-accountingservice       1/1     1            1           23m
deployment.apps/otel-demo-adservice               1/1     1            1           23m
deployment.apps/otel-demo-cartservice             1/1     1            1           23m
deployment.apps/otel-demo-checkoutservice         1/1     1            1           23m
deployment.apps/otel-demo-currencyservice         1/1     1            1           23m
deployment.apps/otel-demo-emailservice            1/1     1            1           23m
deployment.apps/otel-demo-flagd                   1/1     1            1           23m
deployment.apps/otel-demo-frauddetectionservice   1/1     1            1           23m
deployment.apps/otel-demo-frontend                1/1     1            1           23m
deployment.apps/otel-demo-frontendproxy           1/1     1            1           23m
deployment.apps/otel-demo-grafana                 1/1     1            1           23m
deployment.apps/otel-demo-imageprovider           1/1     1            1           23m
deployment.apps/otel-demo-jaeger                  1/1     1            1           23m
deployment.apps/otel-demo-kafka                   1/1     1            1           23m
deployment.apps/otel-demo-loadgenerator           1/1     1            1           23m
deployment.apps/otel-demo-otelcol                 1/1     1            1           23m
deployment.apps/otel-demo-paymentservice          1/1     1            1           23m
deployment.apps/otel-demo-productcatalogservice   1/1     1            1           23m
deployment.apps/otel-demo-prometheus-server       1/1     1            1           23m
deployment.apps/otel-demo-quoteservice            1/1     1            1           23m
deployment.apps/otel-demo-recommendationservice   1/1     1            1           23m
deployment.apps/otel-demo-shippingservice         1/1     1            1           23m
deployment.apps/otel-demo-valkey                  1/1     1            1           23m

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/otel-demo-accountingservice-996db44fd        1         1         1       23m
replicaset.apps/otel-demo-adservice-6d9788b57d               1         1         1       23m
replicaset.apps/otel-demo-cartservice-7cff65c78f             1         1         1       23m
replicaset.apps/otel-demo-checkoutservice-799bfd4698         1         1         1       23m
replicaset.apps/otel-demo-currencyservice-6b48b4d447         1         1         1       23m
replicaset.apps/otel-demo-emailservice-9c6cdbdd8             1         1         1       23m
replicaset.apps/otel-demo-flagd-dd64896b                     1         1         1       23m
replicaset.apps/otel-demo-frauddetectionservice-797598cc59   1         1         1       23m
replicaset.apps/otel-demo-frontend-f9d55f999                 1         1         1       23m
replicaset.apps/otel-demo-frontendproxy-587fd84d44           1         1         1       23m
replicaset.apps/otel-demo-grafana-75f688869c                 1         1         1       23m
replicaset.apps/otel-demo-imageprovider-cb8c66b4c            1         1         1       23m
replicaset.apps/otel-demo-jaeger-5f44f99657                  1         1         1       23m
replicaset.apps/otel-demo-kafka-6c4cfcb854                   1         1         1       23m
replicaset.apps/otel-demo-loadgenerator-5944bf844c           1         1         1       23m
replicaset.apps/otel-demo-otelcol-7596b9ffd7                 1         1         1       23m
replicaset.apps/otel-demo-paymentservice-7b884d56dd          1         1         1       23m
replicaset.apps/otel-demo-productcatalogservice-5959cd684c   1         1         1       23m
replicaset.apps/otel-demo-prometheus-server-78869b8654       1         1         1       23m
replicaset.apps/otel-demo-quoteservice-bfb7cb5c6             1         1         1       23m
replicaset.apps/otel-demo-recommendationservice-67f8fff94c   1         1         1       23m
replicaset.apps/otel-demo-shippingservice-75c98bd59f         1         1         1       23m
replicaset.apps/otel-demo-valkey-668fdfcffc                  1         1         1       23m

NAME                                    READY   AGE
statefulset.apps/otel-demo-opensearch   1/1     23m
ronanorm@rnws-linK|opentelemetry-demo$>> kubectl get ingress
NAME                      CLASS     HOSTS                    ADDRESS       PORTS   AGE
otel-demo-frontendproxy   traefik   otel-demo-lcl.home.lab   10.10.10.25   80      24m

[julianocosta89]

Hello @Ron-Norman 👋🏽

Thanks for reporting this.

We have updated the helm chart last week, are you using the latest version?

[puckpuck]

I wonder if setting the service type to LoadBalancer on frontendProxy is the culprit here. All data including OTLP telemetry goes through it.

Can you try to deploy without the ingress route and see if it works?

[puckpuck]

I've been able to replicate this on my end. It was due to this PR, which upgraded the Prometheus version to 3.x and made a few breaking changes with the OTLP receiver.

I have a fix tested and ready to go.