CSV deployment annotations not showing in final deployment/pods

[groeges]

Bug Report

Any deployment template annotations defined in a CSV do not get included in the deployment when the operator is installed. Not sure whether this is a bug or something that is not currently possible. Any help/guidance on providing annotation for the deployment template would be greatly received.

What did you do?
Generated a CSV (not full csv definition show here) that contained some annotations,
spec.install.spec.deployments.spec.template.metadata.annotations

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    capabilities: Basic Install
    operators.operatorframework.io/builder: operator-sdk-v1.0.0
    operators.operatorframework.io/project_layout: go.kubebuilder.io/v2
  name: my-operator.v0.0.1
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
      - description: Elasticsearch is the Schema for the elasticsearches API
        displayName: Elasticsearch
        kind: Elasticsearch
        name: elasticsearches.elastic.com
        version: v1alpha1
  description: my operator
  displayName: my-operator
  icon:
    - base64data: ""
      mediatype: ""
  install:
    spec:
      clusterPermissions:
        - rules:
            - apiGroups:
                - apps
              resources:
                - deployments
              verbs:
                - get
      deployments:
        - name: mydeployment
          spec:
            replicas: 1
            selector:
              matchLabels:
                app.kubernetes.io/component: elasticsearch
                control-plane: controller-manager
            strategy: {}
            template:
              metadata:
                annotations:
                  myAnnnotation1: ALL
                  myAnnotation2: none
                labels:
                  app.kubernetes.io/component: mycomponent
                  control-plane: controller-manager
              spec:
.....

What did you expect to see?
When the operator was installed I expected to see the

   myAnnotation1: ALL
   myAnnotation2: none

annotations on the "Deployment" under spec.template.metadata.anotations, for example:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    deployment.kubernetes.io/revision: '2'
  resourceVersion: '25276567'
  name: mydeployment
  uid: 2aba1f32-5ae2-4854-9dca-500820a14900
  creationTimestamp: '2020-11-18T10:41:06Z'
  generation: 2
  managedFields: {}
  namespace: default
  labels:
    app.kubernetes.io/component: elasticsearch
    certmanager.k8s.io/time-restarted: 2020-11-18.1041
    control-plane: controller-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: elasticsearch
      control-plane: controller-manager
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: elasticsearch
        certmanager.k8s.io/time-restarted: 2020-11-18.1041
        control-plane: controller-manager
      annotations:
        myAnnnotation1: ALL
        myAnnotation2: none
    spec:
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 10
      securityContext:
        runAsNonRoot: true
      containers:
 ....

What did you see instead? Under which circumstances?
However, when the operator was installed the Deployment did not have these values.

Should I expect these annotations to be available on the Deployment.
If not, how would I get these annotations on the deployment

Environment

• operator-lifecycle-manager version:

• Kubernetes version information:

• Kubernetes cluster kind:

Possible Solution

Additional context
Applying CSV and installing operator on RedHat OpenShift 4.6

[exdx]

Hi @groeges,

I assume the annotations are not present on the deployment pods as well? We definitely want to pass the annotations down from the CSV spec to the deployment.

Here's where we extract the annotations from the CSV when creating the deployment:
https://github.com/operator-framework/operator-lifecycle-manager/blame/master/pkg/controller/install/deployment.go#L135

I believe @awgreene may have looked into a related bug recently? I don't see why here the annotations wouldn't be propagated downwards.

[groeges]

Hi @exdx,

Thanks for looking into this. I can confirm that the annotations do not appear on the ReplicaSet or the Pods.

[exdx]

We looked into this a bit and were unable to reproduce in a local minikube cluster or on an OpenShift 4.6.1 cluster.

Could you try reproducing this locally via cloning OLM and running make run-local on a running minikube cluster and see if it still is occuring?

[groeges]

Having done a bit more investigation from our side. Our CSV also has a webhookdefinition for a ValidatingAdmissionWebhook, for example:

  webhookdefinitions:
  - admissionReviewVersions:
    - v1beta1
    containerPort: 443
    deploymentName: mydeployment
    failurePolicy: Fail
    generateName: velasticsearch.kb.io
    rules:
    - apiGroups:
      - elastic.com
      apiVersions:
      - v1alpha1
      operations:
      - CREATE
      - UPDATE
      resources:
      - elasticsearches
    sideEffects: None
    targetPort: 9443
    type: ValidatingAdmissionWebhook
    webhookPath: /validate-elastic-com-v1alpha1-elasticsearch

If we remove the webhookdefinition from the CSV the annotations appear correctly in the Deployment,ReplicaSet and Pods but with the webhookdefinition these do not appear.

[groeges]

Adding the annotations to the CSV metadata.annotations allows this to work with the webhookdefinition specified.
We should not have to add the annotations to the CSV metadata.annotations but this can be used as a workaround!

[exdx]

Having done a bit more investigation from our side. Our CSV also has a webhookdefinition for a ValidatingAdmissionWebhook, for example:

  webhookdefinitions:
  - admissionReviewVersions:
    - v1beta1
    containerPort: 443
    deploymentName: mydeployment
    failurePolicy: Fail
    generateName: velasticsearch.kb.io
    rules:
    - apiGroups:
      - elastic.com
      apiVersions:
      - v1alpha1
      operations:
      - CREATE
      - UPDATE
      resources:
      - elasticsearches
    sideEffects: None
    targetPort: 9443
    type: ValidatingAdmissionWebhook
    webhookPath: /validate-elastic-com-v1alpha1-elasticsearch

If we remove the webhookdefinition from the CSV the annotations appear correctly in the Deployment,ReplicaSet and Pods but with the webhookdefinition these do not appear.

Interesting, thanks for finding that. Seems like we are overwriting the annotations when a webhook defined somewhere. Will look into it.

[exdx]

Closing this as I believe #1913 addresses the issue. If not, feel free to reopen.