Helm upgrade fails to install clustertriggerauthentications crd #139

stefan-marc · 2021-03-26T15:08:07Z

Report

Hi,
Not sure if this is the correct place for this however, here we go...
Helm upgrade from 2.0.0 to 2.2.0 fails to install (clustertriggerauthentications.keda.sh crd)
We're running keda v2.0.0 installed via helm 3 on v1.19.6 (AKS)

Installed using --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity>

Since the crd is missing pod fails to start.
(no matches for kind "ClusterTriggerAuthentication" in version "keda.sh/v1alpha1)

Expected Behavior

Missing components are installed and images are upgraded to the version specified.

Actual Behavior

Images get upgraded to the version specified.
Missing components are not installed, in this case clustertriggerauthentications.keda.sh is not installed.

If I manually install the missing crd, everything works as expected.

Steps to Reproduce the Problem

Ensure that keda crds are not already installed
helm install keda kedacore/keda -n keda --version=2.0.0 --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity> --debug
helm upgrade helm install keda kedacore/keda -n keda --version=2.2.0 --debug
check pod logs, check crd

Logs from KEDA operator

2021-03-26T14:47:51.962Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-03-26T14:47:51.965Z	INFO	controllers.ScaledObject	Running on Kubernetes 1.19	{"version": "v1.19.6"}
2021-03-26T14:47:51.965Z	INFO	setup	Starting manager
2021-03-26T14:47:51.965Z	INFO	setup	KEDA Version: 2.2.0
2021-03-26T14:47:51.965Z	INFO	setup	Git Commit: 6dc4c3524ba6278276cd1338cc7e4e9a7dd88963
2021-03-26T14:47:51.965Z	INFO	setup	Go Version: go1.15.6
2021-03-26T14:47:51.965Z	INFO	setup	Go OS/Arch: linux/amd64
I0326 14:47:51.966071       1 leaderelection.go:243] attempting to acquire leader lease keda/operator.keda.sh...
2021-03-26T14:47:51.966Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I0326 14:48:09.381707       1 leaderelection.go:253] successfully acquired lease keda/operator.keda.sh
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.482Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-03-26T14:48:10.285Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "ClusterTriggerAuthentication.keda.sh", "error": "no matches for kind \"ClusterTriggerAuthentication\" in version \"keda.sh/v1alpha1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:184
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/manager/internal.go:676
2021-03-26T14:48:10.285Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "worker count": 1}
2021-03-26T14:48:10.285Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}

KEDA Version

2.2.0

Kubernetes Version

1.19

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

The text was updated successfully, but these errors were encountered:

herrbpl · 2021-04-07T11:38:40Z

Have same issue

tomkerkhove · 2021-04-08T06:38:27Z

Might be related to helm/helm#6581 & helm/helm#8668

tomkerkhove · 2021-04-08T06:42:28Z

As a mitigation, you can manually install the CRD (https://github.com/kedacore/charts/blob/master/keda/crds/clustertriggerauthentications.yaml).

tomkerkhove · 2021-04-08T06:47:22Z

As per: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#install-a-crd-declaration-before-using-the-resource

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

It looks like we'll need to find a way around this.

MattJeanes · 2021-04-08T14:33:41Z

This should be noted in upgrade notes as when installing through Helm this is technically a breaking change

Workaround: kubectl apply -f https://raw.githubusercontent.com/kedacore/charts/master/keda/crds/clustertriggerauthentications.yaml

tomkerkhove · 2021-04-11T09:02:14Z

I've opened #140 to fix this so that they are always added and I'll update the docs asap.

Sorry for the inconvenience. I wish Helm was handling this better.

stefan-marc · 2021-04-12T10:36:32Z

Hi @tomkerkhove,

Hope you're well.

I can confirm the workaround works since I tested it before creating the ticket.

However, in this specific case we're not actually talking about a CRD upgrade using Helm since the actual CRD is missing to begin with. So my exception would be that if the CRD is not installed and keda has a hard dependency on it, exactly the situation we're facing now , it should be installed.

I would also argue that the user should have the possibility of forcing the upgrade of a CRD through Helm, using a specific switch (with full accountability of course on his end, knowing that the process is not supported and may potentially cause data lose).

Anyway that's just my point of view.

Let me know what you think.

Thanks for all your help and support.
Be safe!

tomkerkhove · 2021-04-12T11:08:15Z

Fully agree, but apparently Helm doesn't agree on that (see above).

I'm working on a fix for this in #141

tomkerkhove · 2021-04-14T17:29:10Z

This is fixed and available: https://github.com/kedacore/charts/releases/tag/v2.2.1

Sorry about this.

micaelgallego · 2021-04-17T08:31:11Z

Install doc still references to this issue. But I assume this issue is fixed and workaround is not longer needed, right?

tomkerkhove · 2021-04-17T09:38:14Z

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

maxisam · 2021-05-03T03:39:17Z

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

2.2 still points to here

tomkerkhove · 2021-05-03T09:07:21Z

Thanks for letting me know @maxisam!

tomkerkhove transferred this issue from kedacore/keda Apr 8, 2021

tomkerkhove mentioned this issue Apr 11, 2021

Move the CRDs from crds/ to templates/ to maintain them #140

Closed

This was referenced Apr 12, 2021

docs: Add note on CRD + Helm installation kedacore/keda-docs#418

Merged

fix: Automatically manage CRDs through Helm deployments #141

Merged

tomkerkhove closed this as completed Apr 14, 2021

tomkerkhove mentioned this issue May 3, 2021

docs: Remove note on CRD + Helm installation for KEDA v2.2 kedacore/keda-docs#432

Merged

shakaib-arif mentioned this issue May 5, 2022

Getting "no matches for kind "ScaledObject"" kedacore/keda#1279

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm upgrade fails to install clustertriggerauthentications crd #139

Helm upgrade fails to install clustertriggerauthentications crd #139

stefan-marc commented Mar 26, 2021

herrbpl commented Apr 7, 2021

tomkerkhove commented Apr 8, 2021 •

edited

Loading

tomkerkhove commented Apr 8, 2021

tomkerkhove commented Apr 8, 2021

MattJeanes commented Apr 8, 2021

tomkerkhove commented Apr 11, 2021

stefan-marc commented Apr 12, 2021

tomkerkhove commented Apr 12, 2021

tomkerkhove commented Apr 14, 2021

micaelgallego commented Apr 17, 2021

tomkerkhove commented Apr 17, 2021

maxisam commented May 3, 2021

tomkerkhove commented May 3, 2021

Helm upgrade fails to install clustertriggerauthentications crd #139

Helm upgrade fails to install clustertriggerauthentications crd #139

Comments

stefan-marc commented Mar 26, 2021

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

herrbpl commented Apr 7, 2021

tomkerkhove commented Apr 8, 2021 • edited Loading

tomkerkhove commented Apr 8, 2021

tomkerkhove commented Apr 8, 2021

MattJeanes commented Apr 8, 2021

tomkerkhove commented Apr 11, 2021

stefan-marc commented Apr 12, 2021

tomkerkhove commented Apr 12, 2021

tomkerkhove commented Apr 14, 2021

micaelgallego commented Apr 17, 2021

tomkerkhove commented Apr 17, 2021

maxisam commented May 3, 2021

tomkerkhove commented May 3, 2021

tomkerkhove commented Apr 8, 2021 •

edited

Loading