OLM Fails to Install 2nd Controller if Any Other Controller is Present within a Cluster

[acornett21]

Describe the bug
We have been testing both the S3 and Elasticache controllers in an OpenShift Cluster
and we ran into an issue where if one controller is installed, and we try to install a second controller, OLM fails to install the second controller.
If we only install one controller at a time S3 or Elasticache OLM installs the controller successfully. And then the controller is able to
manage it’s CustomResources successfully.
We have pinpointed it to the AdoptedResource custom resource. We are noticed that across all the controllers the AdoptedResource’s
apiVersion: adoptedresources.services.k8s.aws/v1alpha1 which causes a collision in OLM, and the second operator to fail the install.
This would mean that only one controller would be able to be installed on a cluster at a time.
Taking a further look, we noticed that the proposed feature request doc
for AdoptedResource that the apiVesion for each controller is proposed as being apiVersion: s3.services.k8s.aws/v1alpha1
and apiVersion: apigateway.services.k8s.aws/v1alpha1 for example. Is the proposed doc yet to be implemented? Or is the plan
to go is the group for AdoptedResource going to be the same across all controllers?

Steps to reproduce

1. Build and bundle two controllers with ACK_GENERATE_OLM=true
2. Try to install both via OLM
3. The 2nd controller never completely gets installed

Expected outcome
Would expect that multiple controllers to be able to be installed within an OpenShift cluster

Environment

• Kubernetes version v1.21
• Using EKS (yes/no), if so version? no
• AWS service targeted (S3, RDS, etc.) - all

[RedbackThomson]

As per our conversation on Slack:

Yes I think this was an oversight in the original proposal. Since the spec for all adopted resources is the same for any resource, I chose to implement it as a common CRD between all services - and hence a common API group. I was aware that the CRDs would collide, but hoped that this did not affect anything.

Helm has a whole page about how it handles CRDs - https://helm.sh/docs/chart_best_practices/custom_resource_definitions/ . We don't recommend installing CRDs using Helm, because there is no way to update them when a new API version is released. Hence all of our documentation and testing scripts install CRDs separately, then use --skip-crds for the charts.

All controllers use a shared informer, filtering based on the spec.kubernetes.group field

[RedbackThomson]

Since OLM generates a separate bundle, using the Kustomize files we generate for the normal installer, it is possible to leave out the crds/common directory, which contains the AdoptedResource CRD. What are the capabilities of the OLM post-install scripts? Is it capable of applying a K8s manifest to the cluster?

[acornett21]

@RedbackThomson Since our script is downstream of the main build scripts, and the OLM generations works off of a temp directory we will probably end up taking a similar approach to leaving out that crd/common director, so kustomize does not pick up that file.

[acornett21]

/assign

[RedbackThomson]

Summary of our conversations:

• The OLM team will modify the installers to remove the AdoptedResource CRD by default, providing documentation for enabling after the fact
• I will work on modifying the controller to detect the presence (or lack) of the CRD, and enable the adoption reconciler only if it exists (Argo reference implementation)
• There should be no functional changes to the upstream ACK controllers

[acornett21]

@RedbackThomson Now that your PR is merged should we close this issue? I don't think there is anything else outstanding.

[RedbackThomson]

@RedbackThomson Now that your PR is merged should we close this issue? I don't think there is anything else outstanding.

Sure! If you are happy that this solution solves your problem, lets close this issue.