bitnami/kube-prometheus fails to deploy

[sudhirpandey]

Name and Version

bitnami/kube-prometheus

What steps will reproduce the bug?

We are using helmfile to deploy the charts. We aare trying to apply we get the error, but when we use helmfile to invoke helm template it generates the manifets "helmfile -e sit -l name=prometheus-thanos-sidecar template"

1. have this in values.yaml

  - name: prometheus-thanos-sidecar
    chart: bitnami/kube-prometheus
    namespace: {{ .Environment.Name }}-prometheus
    values:
    - prometheus-values.yaml.gotmpl

in prometheus-values.yaml.gotmpl we have

alertmanager:
  enabled: false

2. try to deploy the chart

 helmfile -e sit -l name=prometheus-thanos-sidecar apply

we get error as below

helmfile -e sit -l name=prometheus-thanos-sidecar apply
Adding repo thanos https://charts.bitnami.com/bitnami
"thanos" has been added to your repositories

Adding repo ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver/
"ebs-csi-driver" has been added to your repositories

Comparing release=prometheus-thanos-sidecar, chart=bitnami/kube-prometheus
in ./helmfile.yaml: command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ARGS:
  0: helm (4 bytes)
  1: diff (4 bytes)
  2: upgrade (7 bytes)
  3: --reset-values (14 bytes)
  4: --allow-unreleased (18 bytes)
  5: prometheus-thanos-sidecar (25 bytes)
  6: bitnami/kube-prometheus (23 bytes)
  7: --namespace (11 bytes)
  8: sit-prometheus (14 bytes)
  9: --values (8 bytes)
  10: /var/folders/d5/n1g_pz216h1g7l3mzjr897pr0000gn/T/helmfile1426978678/sit-prometheus-prometheus-thanos-sidecar-values-798465fd84 (126 bytes)
  11: --detailed-exitcode (19 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  reading three way merge from env
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]
  Error: plugin "diff" exited with error

COMBINED OUTPUT:
  reading three way merge from env
  ********************
  	Release was not present in Helm.  Diff will show entire contents as new.
  ********************
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]
  Error: plugin "diff" exited with error

Unsure if we need to deploy the CRs in cluster before we do this helm deployment.

Are you using any custom parameters or values?

values.yaml

alertmanager:
  enabled: false

What is the expected behavior?

We expect it to deploy the chart

What do you see instead?

erro as below
STDERR:
reading three way merge from env
Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]
Error: plugin "diff" exited with error

COMBINED OUTPUT:
reading three way merge from env

---

Release was not present in Helm.  Diff will show entire contents as new.

---

Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]
Error: plugin "diff" exited with error

Additional information

No response

[sudhirpandey]

it seems helm does not roll out the necessary crds hence these need to apply first

❯ kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

❯ kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml

so probably would be nice to get it added in Read.me as pre req

[juan131]

Hi @sudhirpandey

I'm afraid I'm not familiar with helmfile. Is it based on Helm2 or Helm3? Please note that this chart is meant for Helm3 as it's listed in the prerequisites section, see:

• https://github.com/bitnami/charts/tree/master/bitnami/kube-prometheus#prerequisites

The CRDs are not including in the "templates" folder and they're under the crds folder as recommended by Helm, see:

• https://helm.sh/docs/chart_best_practices/custom_resource_definitions

[sudhirpandey]

Hi @juan131

Helmfile is based on helm3 for me at least ,. as i have helm3 installed

helm version
version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.17.3"}

more docs here https://github.com/roboll/helmfile.

Great the crds are here
https://github.com/bitnami/charts/tree/master/bitnami/kube-prometheus/crds

but when we apply since these are not in templates , how are these crds deployed ?
in a similar chart like this one , https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

in readme it specifies "Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: "

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-19x-to-20x

Hence that helps a lot to understand what is needed before you attempted to deploy the chart.

If helm was suppose to deploy the crds then potentally i am running into a bug ??, but not sure where it comes from.

[sudhirpandey]

okey i think i am running into this issue roboll/helmfile#1124 :).

[juan131]

Hi @sudhirpandey

but when we apply since these are not in templates , how are these crds deployed ?

They're deployed as pure YAML manifests during the chart installation and they're ignored during Helm upgrades.

okey i think i am running into this issue roboll/helmfile#1124 :)

Were you able to solve the issue following the solutions proposed in that thread?

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.