Cannot install prometheus-operator chart with helmfile due to helm-diff error #1124

mojochao · 2020-02-26T19:48:20Z

Versions used:

helmfile v0.100.2
helm v3.1.1
databus23/helm-diff plugin v3.1.1

Problem observed:

I can successfully install the stable/prometheus-operator chart with helm directly.

$ helm install prometheus-operator stable/prometheus-operator -n monitoring
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: prometheus-operator
LAST DEPLOYED: Wed Feb 26 13:13:58 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus Operator has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus-operator"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

I cannot install the same chart with helmfile:

$ helmfile apply
Comparing release=prometheus-operator, chart=stable/prometheus-operator
in ./helmfile.yaml: in .helmfiles[0]: in helmfiles/prometheus-operator.yaml: failed processing release prometheus-operator: helm exited with status 1:
  Error: Failed to render chart: exit status 1: manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

  Error: plugin "diff" exited with error

I see that the error is coming out of the helm-diff plugin. What command(s) is helmfile using that causes these errors to surface? I will open an issue in the databus23/helm-diff plugin repo once I understand how to repro this with the helm diff command directly.

Many thanks in advance!

The text was updated successfully, but these errors were encountered:

mojochao · 2020-02-26T20:01:28Z

I made an educated guess as to the helm diff command:

$ helm diff --allow-unreleased upgrade -n monitoring prometheus-operator stable/prometheus-operator
********************

	Release was not present in Helm.  Diff will show entire contents as new.

********************
Error: Failed to render chart: exit status 1: manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

Error: plugin "diff" exited with error

Hopefully that command is in the ballpark of the real command. My guess is that until the helm diff command succeeds, helmfile won't be able to successfully install that chart. :-) I'll report the Issue in their repo.

mojochao · 2020-02-26T20:08:31Z

Issue being tracked in databus23/helm-diff repo with databus23/helm-diff#189

andrewnazarov · 2020-02-27T11:16:47Z

What version of prometheus-operator are you trying to install?

andrewnazarov · 2020-02-27T11:19:37Z

Probably your version is a bit outdated. Helm 3 works differently with CRDs: https://helm.sh/docs/topics/chart_best_practices/custom_resource_definitions/

mojochao · 2020-02-27T20:45:32Z

I'm using helm v3.1.1, helm-diff plugin v3.1.1 trying to install `stable/prometheus-operator` chart. Does this chart not properly handle CRDs? Is there a different chart I should be using? Thanks for the assistance, Andrew!

…

On Thu, Feb 27, 2020 at 5:19 AM Andrew Nazarov ***@***.***> wrote: Probably your version is a bit outdated. Helm 3 works differently with CRDs: https://helm.sh/docs/topics/chart_best_practices/custom_resource_definitions/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1124?email_source=notifications&email_token=AAA2PMPIXVCGMEHUF6VAAATRE6OUVA5CNFSM4K4MVLUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND7MNI#issuecomment-591918645>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2PMK5DZYKD7MNSZ6IFALRE6OUVANCNFSM4K4MVLUA> .

andrewnazarov · 2020-02-29T12:16:36Z

We are successfully deploying version 8.7.0 with Helm 3.

Can you check the exact version of the prometheus-operator chart that is used?

imtpot · 2020-03-24T16:38:45Z

I'm using hooks to workaround

hooks:
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
        kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
        kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml"]  
  set:
  - name: prometheusOperator.createCustomResource
    value: false

mojochao · 2020-03-25T20:28:10Z

With helmfile (v0.104.1), helm-diff (v3.1.1), and helm (v3.1.2) I'm no longer having this issue.

…

On Tue, Mar 24, 2020 at 11:39 AM agmtr ***@***.***> wrote: I'am use hooks to workaround - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml"] set: - name: prometheusOperator.createCustomResource value: false``` — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2PMIESYDOZ5SLQBIVS6LRJDO2LANCNFSM4K4MVLUA> .

rchenzheng · 2020-03-30T15:09:10Z

This is still happening as of helmfile (v0.106.3), helm-diff (v3.1.1), and helm (v3.1.2)

What prometheus-operator chart version are you using ?@mojochao

I seem to have the same issue with 8.12.3

giannisbetas · 2020-04-17T16:57:31Z

helmfile: v0.109.0
helm: v3.1.2
helm-diff: 3.1.1

Chart:
prometheus-operator: 8.12.13

The issue is not solved.
In order to work around it I had to install the CRDs manually

logston · 2020-04-24T09:04:58Z

Similar. But a different chart.

Bash:

$ helmfile --version
helmfile version v0.109.1
$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
$ helm plugin list
NAME    VERSION DESCRIPTION                           
diff    3.1.1   Preview helm upgrade changes as a diff

helmfile:

repositories:
- name: dynatrace
  url: https://raw.githubusercontent.com/Dynatrace/helm-charts/master/repos/stable

releases:
  - name: dynatrace-oneagent-operator
    namespace: dynatrace
    chart: dynatrace/dynatrace-oneagent-operator
    version: ~0.7.0

mumoshu · 2020-04-24T09:25:05Z

Isn't this chart issue, rather than helm or helmfile's?

I mean, are those charts migrated to internally use the new crds directory added in Helm 3?

Xartos · 2020-04-24T13:59:06Z

I would rather say a helm 3 issue. Since doing helm install --dry-run don't work for charts with crds, see helm/helm#7449
And in the end that's why this fails as well, you can't verify objects that aren't yet defined in the API-server.

In my opinion the best solution would be the one suggested in that issue, but it doesn't seem to be a priority atm. So another solution would be to have a way to disable diff in helmfile when the release aren't present (when installing), like a --skip-diff-if-installing flag when doing apply or something.

mumoshu · 2020-04-24T14:22:55Z

Thanks. Interesting idea!

Just curious, but do you think how could Helmfile determines if it's an install or upgrade?

Xartos · 2020-04-24T14:26:17Z

I don't know how you would do it optimally, but I guess it could do helm list and see if the release exists or something similar.

andrewnazarov · 2020-04-24T14:30:05Z

But does helmfile ever do helm install --dry-run?
helm-diff plugin does helm upgrade --debug --dry-run

Xartos · 2020-04-24T14:36:54Z

It probably don't. The point I was trying to make is that the root cause is the same. As long as you try to validate an object that is not defined in the API-server, the validation will fail.

It can't be running that for an install right, atleast not without the --install flag?
I know that helm-diff runs helm template ... --validate which also validates the rendered manifests to the API-server and thus the same error occurs.

baurmatt · 2020-05-11T15:22:04Z

We currently using the following workaround:

...
    hooks:
    # Create CRDs separately in helmfile presync hooks
    # https://github.com/roboll/helmfile/issues/1124
    # https://github.com/helm/helm/issues/7449
    # https://github.com/cloudposse/helmfiles/blob/59490fd2599d6113a14103be919985f9fbcea73a/releases/prometheus-operator.yaml
    # Hooks associated to presync events are triggered before each release is applied to the remote cluster.
    # This is the ideal event to execute any commands that may mutate the cluster state as it
    # will not be run for read-only operations like lint, diff or template.
    # These hook install the prometheuses.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-prometheus.yaml | kubectl apply -f -; }"]
    # This hoook installs the alertmanagers.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-alertmanager.yaml | kubectl apply -f -; }"]
    # This hoook installs the prometheusrules.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-prometheusrules.yaml | kubectl apply -f -; }"]
    # This hoook installs the servicemonitors.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-servicemonitor.yaml | kubectl apply -f -; }"]
    # This hoook installs the podmonitors.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-podmonitor.yaml | kubectl apply -f -; }"]
    # This hoook installs the thanosrulers.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd thanosrulers.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-thanosrulers.yaml | kubectl apply -f -; }"]
...

It's similar to the one posted above by @agmtr but uses the CRDs directly from the chart. This increases compatibility because it uses the same version which is normally bundled with the chart - just deploys it with a presync hook.

semoac · 2020-05-19T17:42:07Z

As a workaround I installed helm-diff 3.0.0-rc.7

Barna001 · 2020-06-04T11:59:55Z

I had the exact same problem, used "8.13.12" version for prometheus-operator and it worked :)
Helm 3 does not support crd-install hook anymore.

bradenwright · 2020-07-01T07:59:49Z

FWIW using helmfile sync works, but using sync skips the diff. Right now I've been using sync on my first install and diff/apply after that.

mumoshu · 2020-07-20T01:23:35Z

I thought helm-diff recently added --disable-openapi-validation that can be used for disabling validation at all which will also makes this issue disappear.

Can we probably enhance Helmfile to add a new option under releases[] to enable it? e.g.

releases:
- name: prometheus-operator
  chart: stable/prometheus-operator
  # Set this to true if your chart contains crds or install-crd hooks.
  disableOpenAPIValidation: true

mumoshu · 2020-07-20T01:23:43Z

databus23/helm-diff#220

mumoshu · 2020-07-22T13:55:38Z

Apparently --disable-openapi-validation doesn't work(It can be a solution for other problems)

But I've managed to make prometheus-operator installation work with --disable-validation I've added to helm-diff today

databus23/helm-diff#228

`disableOpenAPIValidation: true` might be useful for workaround for broken CRDs that is known to be exist in older OpenShift versions, and `disableValidation: true` is confirmed to allow installing charts like prometheus-operator that tries to install CRDs and CRs in the same chart. Strictly speaking, for the latter case I believe you only need `disableValidation: true` set during the first installation, but for the ease of operation I shall suggest you to always set it. Obviously turning validation mostly(disableOpenAPIValidation) or entirely(disableValidation) result in deferring any real error until sync time. We need completely client-side validation that is able to read CRDs and use it for validating any CRs to catch any error before sync. But it worth an another (big) issue. Fixes #1124

mumoshu · 2020-07-22T14:11:49Z

So after #1373 this should work

releases:
- name: prometheus-operator
  chart: stable/prometheus-operator
  disableValidation: true

Xtigyro · 2021-02-03T16:09:44Z

@mumoshu Thank you, man - as always you rock!!!

mojochao changed the title ~~Cannot install prometheus-operator chart with helmfile~~ Cannot install prometheus-operator chart with helmfile due to helm-diff error Feb 26, 2020

andrewnazarov mentioned this issue Jul 13, 2020

helmfile apply does not install crds folder #1353

Closed

lukasmrtvy mentioned this issue Jul 17, 2020

hooks not called with destroy #802

Closed

mumoshu mentioned this issue Jul 22, 2020

Add disableValidation and disableOpenAPIValidation per release #1373

Merged

mumoshu closed this as completed in #1373 Jul 22, 2020

jkroepke mentioned this issue Nov 24, 2020

Allow helm diff failures #1600

Open

sudhirpandey mentioned this issue Mar 3, 2022

bitnami/kube-prometheus fails to deploy bitnami/charts#9270

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot install prometheus-operator chart with helmfile due to helm-diff error #1124

Cannot install prometheus-operator chart with helmfile due to helm-diff error #1124

mojochao commented Feb 26, 2020

mojochao commented Feb 26, 2020

mojochao commented Feb 26, 2020

andrewnazarov commented Feb 27, 2020

andrewnazarov commented Feb 27, 2020

mojochao commented Feb 27, 2020 via email

andrewnazarov commented Feb 29, 2020

imtpot commented Mar 24, 2020 •

edited

Loading

mojochao commented Mar 25, 2020 via email

rchenzheng commented Mar 30, 2020 •

edited

Loading

giannisbetas commented Apr 17, 2020

logston commented Apr 24, 2020 •

edited

Loading

mumoshu commented Apr 24, 2020 •

edited

Loading

Xartos commented Apr 24, 2020

mumoshu commented Apr 24, 2020

Xartos commented Apr 24, 2020

andrewnazarov commented Apr 24, 2020

Xartos commented Apr 24, 2020

baurmatt commented May 11, 2020

semoac commented May 19, 2020

Barna001 commented Jun 4, 2020 •

edited

Loading

bradenwright commented Jul 1, 2020

mumoshu commented Jul 20, 2020

mumoshu commented Jul 20, 2020

mumoshu commented Jul 22, 2020

mumoshu commented Jul 22, 2020

Xtigyro commented Feb 3, 2021

Cannot install prometheus-operator chart with helmfile due to helm-diff error #1124

Cannot install prometheus-operator chart with helmfile due to helm-diff error #1124

Comments

mojochao commented Feb 26, 2020

mojochao commented Feb 26, 2020

mojochao commented Feb 26, 2020

andrewnazarov commented Feb 27, 2020

andrewnazarov commented Feb 27, 2020

mojochao commented Feb 27, 2020 via email

andrewnazarov commented Feb 29, 2020

imtpot commented Mar 24, 2020 • edited Loading

mojochao commented Mar 25, 2020 via email

rchenzheng commented Mar 30, 2020 • edited Loading

giannisbetas commented Apr 17, 2020

logston commented Apr 24, 2020 • edited Loading

mumoshu commented Apr 24, 2020 • edited Loading

Xartos commented Apr 24, 2020

mumoshu commented Apr 24, 2020

Xartos commented Apr 24, 2020

andrewnazarov commented Apr 24, 2020

Xartos commented Apr 24, 2020

baurmatt commented May 11, 2020

semoac commented May 19, 2020

Barna001 commented Jun 4, 2020 • edited Loading

bradenwright commented Jul 1, 2020

mumoshu commented Jul 20, 2020

mumoshu commented Jul 20, 2020

mumoshu commented Jul 22, 2020

mumoshu commented Jul 22, 2020

Xtigyro commented Feb 3, 2021

imtpot commented Mar 24, 2020 •

edited

Loading

rchenzheng commented Mar 30, 2020 •

edited

Loading

logston commented Apr 24, 2020 •

edited

Loading

mumoshu commented Apr 24, 2020 •

edited

Loading

Barna001 commented Jun 4, 2020 •

edited

Loading