Failed to run bundle: install plan is not available for the subscription

[pohly]

Bug Report

I also reported this in operator-framework/operator-lifecycle-manager#2454 but as this might also be an issue in operator-sdk, let me also file an issue here.

What did you do?

• installed OLM with operator-sdk olm install
• tried to install an operator with operator-sdk run bundle

What did you expect to see?

The operator should start to run.

What did you see instead? Under which circumstances?

operator-sdk run bundle --namespace default --timeout 5m 172.17.42.1:5001/pmem-csi-bundle:v100.0.0 --skip-tls
time="2021-11-17T14:21:35+01:00" level=info msg="Successfully created registry pod: 172-17-42-1-5001-pmem-csi-bundle-v100-0-0"
time="2021-11-17T14:21:35+01:00" level=info msg="Created CatalogSource: pmem-csi-operator-catalog"
time="2021-11-17T14:21:35+01:00" level=info msg="OperatorGroup \"operator-sdk-og\" created"
time="2021-11-17T14:21:35+01:00" level=info msg="Created Subscription: pmem-csi-operator-v100-0-0-sub"
time="2021-11-17T14:26:27+01:00" level=fatal msg="Failed to run bundle: install plan is not available for the subscription pmem-csi-operator-v100-0-0-sub: timed out waiting for the condition\n"

This only happens with OLM 1.19.1. The same commands work when installing OLM 0.18.3 with operator-sdk olm install --version=v0.18.3. UPDATE: there is some randomness involved and it may depend on cluster load and/or state, see #5410 (comment) and #5410 (comment).

Environment

Operator type:

/language go

Kubernetes cluster type:

kubeadm in VMs with Kubernetes 1.21.1

$ operator-sdk version

operator-sdk version: "v1.15.0", commit: "f6326e832a8a5e5453d0ad25e86714a0de2c0fc8", kubernetes version: "1.21", go version: "go1.16.10", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.17.2 linux/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-27T08:53:39Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Additional context

I encountered this in PMEM-CSI, tracked there as intel/pmem-csi#1050

More diagnostics:

$ kubectl describe --all-namespaces subscriptions
Name:         pmem-csi-operator-v100-0-0-sub
Namespace:    default
Labels:       operators.coreos.com/pmem-csi-operator.default=
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         Subscription
Metadata:
  Creation Timestamp:  2021-11-23T15:36:52Z
  Generation:          1
  Resource Version:  3171
  UID:               267d37b7-94c6-428c-a472-eab6ee98289a
Spec:
  Channel:                alpha
  Install Plan Approval:  Manual
  Name:                   pmem-csi-operator
  Source:                 pmem-csi-operator-catalog
  Source Namespace:       default
  Starting CSV:           pmem-csi-operator.v100.0.0
Status:
  Catalog Health:
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              pmem-csi-operator-catalog
      Namespace:         default
      Resource Version:  3164
      UID:               306ec63a-1b78-4fd2-b06e-085194131acf
    Healthy:             true
    Last Updated:        2021-11-23T15:36:52Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              operatorhubio-catalog
      Namespace:         olm
      Resource Version:  3143
      UID:               0ff45726-9642-4187-bff3-3738df547344
    Healthy:             true
    Last Updated:        2021-11-23T15:36:52Z
  Conditions:
    Last Transition Time:  2021-11-23T15:36:52Z
    Message:               all available catalogsources are healthy
    Reason:                AllCatalogSourcesHealthy
    Status:                False
    Type:                  CatalogSourcesUnhealthy
  Last Updated:            2021-11-23T15:36:52Z
Events:                    <none>

Note the odd "AllCatalogSourcesHealthy: False". The catalog-operator pod here might be responsible for it (not sure) and reports an error (E1123 15:36:52.688776 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again):

time="2021-11-23T15:36:32Z" level=info msg="log level info"
time="2021-11-23T15:36:32Z" level=info msg="TLS keys not set, using non-https for metrics"
W1123 15:36:32.456457       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-11-23T15:36:32Z" level=info msg="Using in-cluster kube client config"
time="2021-11-23T15:36:32Z" level=info msg="Using in-cluster kube client config"
W1123 15:36:32.458198       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1123 15:36:32.486566       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-11-23T15:36:32Z" level=info msg="connection established. cluster-version: v1.21.1"
time="2021-11-23T15:36:32Z" level=info msg="operator ready"
time="2021-11-23T15:36:32Z" level=info msg="starting informers..."
time="2021-11-23T15:36:32Z" level=info msg="informers started"
time="2021-11-23T15:36:32Z" level=info msg="waiting for caches to sync..."
time="2021-11-23T15:36:32Z" level=info msg="starting workers..."
time="2021-11-23T15:36:32Z" level=info msg="connection established. cluster-version: v1.21.1"
time="2021-11-23T15:36:32Z" level=info msg="operator ready"
time="2021-11-23T15:36:32Z" level=info msg="starting informers..."
time="2021-11-23T15:36:32Z" level=info msg="informers started"
time="2021-11-23T15:36:32Z" level=info msg="waiting for caches to sync..."
time="2021-11-23T15:36:32Z" level=warning msg="couldn't find service in cache" service=operatorhubio-catalog
time="2021-11-23T15:36:32Z" level=info msg="starting workers..."
time="2021-11-23T15:36:32Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=Fy47U
time="2021-11-23T15:36:32Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=CUJ52
time="2021-11-23T15:36:33Z" level=error msg="UpdateStatus - error while setting CatalogSource status" error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"operatorhubio-catalog\": the object has been modified; please apply your changes to the latest version and try again" id=oXhvn source=operatorhubio-catalog
E1123 15:36:33.394238       1 queueinformer_operator.go:290] sync {"update" "olm/operatorhubio-catalog"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "operatorhubio-catalog": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:33Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=O+6l9
time="2021-11-23T15:36:33Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:33Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=jQGRk
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=QyOmv
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:34Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog"
time="2021-11-23T15:36:35Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=ByiaJ
time="2021-11-23T15:36:36Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:36Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:36:36Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=9+k2z
time="2021-11-23T15:36:36Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=YUXeH
time="2021-11-23T15:36:38Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:38Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:36:38Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=Y6+kZ
time="2021-11-23T15:36:38Z" level=error msg="UpdateStatus - error while setting CatalogSource status" error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"operatorhubio-catalog\": the object has been modified; please apply your changes to the latest version and try again" id=nDl4j source=operatorhubio-catalog
E1123 15:36:38.488617       1 queueinformer_operator.go:290] sync {"update" "olm/operatorhubio-catalog"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "operatorhubio-catalog": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:36:42Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:42Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:36:42Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=H6Zl9
time="2021-11-23T15:36:42Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=1g8ma
time="2021-11-23T15:36:43Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=SrQ8o
time="2021-11-23T15:36:45Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=vkSRo
time="2021-11-23T15:36:45Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=T8OdF
time="2021-11-23T15:36:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:36:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=READY"
time="2021-11-23T15:36:48Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=ZJxsk
time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=sqS0R
time="2021-11-23T15:36:52Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=CONNECTING"
time="2021-11-23T15:36:52Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=READY"
time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=voIzU
time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=JK73i
time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=WDa3B
time="2021-11-23T15:36:52Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:36:52.688776       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=

This repeats a few times but then not anymore. Deleting that pod doesn't help, the recreated one has the same problem.

For comparison, here is the output with OLM 0.18.3. It has the same update error, so that might be a red herring:

Name:         pmem-csi-operator-v100-0-0-sub
Namespace:    default
Labels:       operators.coreos.com/pmem-csi-operator.default=
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         Subscription
Metadata:
  Creation Timestamp:  2021-11-23T15:26:12Z
  Generation:          1
  Managed Fields: [...]
  Resource Version:  1310
  UID:               28bd1f43-0d4b-4f7f-bc3a-ddfbc9e4b080
Spec:
  Channel:                alpha
  Install Plan Approval:  Manual
  Name:                   pmem-csi-operator
  Source:                 pmem-csi-operator-catalog
  Source Namespace:       default
  Starting CSV:           pmem-csi-operator.v100.0.0
Status:
  Catalog Health:
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              pmem-csi-operator-catalog
      Namespace:         default
      Resource Version:  1227
      UID:               7d43c145-9139-4b58-8552-5aa9dc1128be
    Healthy:             true
    Last Updated:        2021-11-23T15:26:24Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              operatorhubio-catalog
      Namespace:         olm
      Resource Version:  1154
      UID:               85a71d8b-ba21-484b-a669-a2fc4d237c8a
    Healthy:             true
    Last Updated:        2021-11-23T15:26:24Z
  Conditions:
    Last Transition Time:   2021-11-23T15:26:24Z
    Message:                all available catalogsources are healthy
    Reason:                 AllCatalogSourcesHealthy
    Status:                 False
    Type:                   CatalogSourcesUnhealthy
    Last Transition Time:   2021-11-23T15:26:23Z
    Reason:                 RequiresApproval
    Status:                 True
    Type:                   InstallPlanPending
  Current CSV:              pmem-csi-operator.v100.0.0
  Install Plan Generation:  1
  Install Plan Ref:
    API Version:       operators.coreos.com/v1alpha1
    Kind:              InstallPlan
    Name:              install-vknrx
    Namespace:         default
    Resource Version:  1209
    UID:               8de30864-1b84-4edb-8425-0f2104649665
  Installed CSV:       pmem-csi-operator.v100.0.0
  Installplan:
    API Version:  operators.coreos.com/v1alpha1
    Kind:         InstallPlan
    Name:         install-vknrx
    Uuid:         8de30864-1b84-4edb-8425-0f2104649665
  Last Updated:   2021-11-23T15:26:26Z
  State:          AtLatestKnown
Events:           <none>



time="2021-11-23T15:25:48Z" level=info msg="log level info"
time="2021-11-23T15:25:48Z" level=info msg="TLS keys not set, using non-https for metrics"
W1123 15:25:48.353944       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-11-23T15:25:48Z" level=info msg="Using in-cluster kube client config"
time="2021-11-23T15:25:48Z" level=info msg="Using in-cluster kube client config"
W1123 15:25:48.355255       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-11-23T15:25:48Z" level=info msg="connection established. cluster-version: v1.21.1"
time="2021-11-23T15:25:48Z" level=info msg="operator ready"
time="2021-11-23T15:25:48Z" level=info msg="starting informers..."
time="2021-11-23T15:25:48Z" level=info msg="informers started"
time="2021-11-23T15:25:48Z" level=info msg="waiting for caches to sync..."
time="2021-11-23T15:25:48Z" level=info msg="starting workers..."
time="2021-11-23T15:25:48Z" level=warning msg="couldn't find service in cache" service=operatorhubio-catalog
time="2021-11-23T15:25:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:25:49Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:25:50Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:25:50Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:25:51Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:25:51Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:25:54Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:25:54Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:25:57Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:25:57Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE"
time="2021-11-23T15:26:04Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING"
time="2021-11-23T15:26:04Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=READY"
time="2021-11-23T15:26:12Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=CONNECTING"
time="2021-11-23T15:26:12Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=READY"
time="2021-11-23T15:26:12Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:13Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:13Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:14Z" level=warning msg="no installplan found with matching generation, creating new one" id=n45Np namespace=default
time="2021-11-23T15:26:14Z" level=warning msg="no installplan found with matching generation, creating new one" id=jqDzy namespace=default
time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:14Z" level=info msg=syncing id=13yW6 ip=install-7k6m5 namespace=default phase=
time="2021-11-23T15:26:14Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=13yW6 ip=install-7k6m5 namespace=default phase=
time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:14Z" level=info msg=syncing id=p6Zl9 ip=install-vknrx namespace=default phase=
time="2021-11-23T15:26:14Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=p6Zl9 ip=install-vknrx namespace=default phase=
time="2021-11-23T15:26:14Z" level=info msg=syncing id=9qY7f ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:14Z" level=warning msg="status not equal, updating..." id=9qY7f ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:15Z" level=info msg=syncing id=eeTWD ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:15Z" level=warning msg="status not equal, updating..." id=eeTWD ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:15Z" level=info msg=syncing id=aG2b8 ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:15Z" level=warning msg="status not equal, updating..." id=aG2b8 ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:16Z" level=info msg=syncing id=JskB/ ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:16.668254       1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:16Z" level=info msg=syncing id=fHgiu ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:16Z" level=warning msg="status not equal, updating..." id=fHgiu ip=install-7k6m5 namespace=default phase=RequiresApproval
I1123 15:26:17.458456       1 request.go:655] Throttling request took 1.174767095s, request: PUT:https://10.96.0.1:443/apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/pmem-csi-operator-v100-0-0-sub/status
time="2021-11-23T15:26:17Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:17.497324       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:17Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:17Z" level=info msg=syncing id=+Eq0P ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:17Z" level=info msg=syncing id=iZBDQ ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:17Z" level=warning msg="status not equal, updating..." id=iZBDQ ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:18Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:18Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:18.474691       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:18Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:18Z" level=info msg=syncing id=aQEtu ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:19Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:19.067953       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:19Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:19Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:19.867193       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:19Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:20Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:20.670721       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:21Z" level=info msg=syncing id=6ZFOt ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=6ZFOt ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:21Z" level=info msg=syncing id=CXnRg ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=CXnRg ip=install-7k6m5 namespace=default phase=RequiresApproval
E1123 15:26:21.665355       1 queueinformer_operator.go:290] sync {"update" "default/install-vknrx"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-vknrx": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:21Z" level=info msg=syncing id=VvKlT ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=VvKlT ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:22.267411       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:22Z" level=info msg=syncing id=6rKkt ip=install-7k6m5 namespace=default phase=RequiresApproval
time="2021-11-23T15:26:22Z" level=info msg=syncing id=XWOvv ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:22Z" level=warning msg="status not equal, updating..." id=XWOvv ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:23.269889       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:23Z" level=info msg=syncing id=WBhSb ip=install-7k6m5 namespace=default phase=RequiresApproval
E1123 15:26:23.671447       1 queueinformer_operator.go:290] sync {"update" "default/install-vknrx"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-vknrx": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:23Z" level=info msg=syncing id=FL6CM ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:23.868491       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:24Z" level=info msg="error updating InstallPlan status" id=WBhSb ip=install-7k6m5 namespace=default phase=RequiresApproval updateError="Operation cannot be fulfilled on installplans.operators.coreos.com \"install-7k6m5\": the object has been modified; please apply your changes to the latest version and try again"
E1123 15:26:24.469377       1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: error updating InstallPlan status: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:24Z" level=info msg=syncing id=6rE5N ip=install-7k6m5 namespace=default phase=Installing
time="2021-11-23T15:26:24Z" level=info msg=syncing id=HYRIR ip=install-vknrx namespace=default phase=RequiresApproval
time="2021-11-23T15:26:24Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:24.869032       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:24Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:25Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
E1123 15:26:25.872468       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:26Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:27Z" level=info msg=syncing id=klg2t ip=install-7k6m5 namespace=default phase=Installing
time="2021-11-23T15:26:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:28Z" level=info msg=syncing id=7Cmjc ip=install-7k6m5 namespace=default phase=Installing
time="2021-11-23T15:26:28Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:28Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml
time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml
time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml
time="2021-11-23T15:26:29Z" level=info msg="error updating InstallPlan status" id=7Cmjc ip=install-7k6m5 namespace=default phase=Installing updateError="Operation cannot be fulfilled on installplans.operators.coreos.com \"install-7k6m5\": the object has been modified; please apply your changes to the latest version and try again"
E1123 15:26:29.471383       1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: error updating InstallPlan status: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-23T15:26:29Z" level=info msg=syncing id=p351h ip=install-7k6m5 namespace=default phase=Complete
time="2021-11-23T15:26:29Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-23T15:26:29Z" level=info msg=syncing id=6j3+Q ip=install-7k6m5 namespace=default phase=Complete

[varshaprasad96]

@pohly do you have prometheus metrics enabled in your operator? If so, is the prometheus stack set up in the cluster. Just checking to see if that is the cause of the issue. Also can you please mention the logs from the installplan if it exists in the cluster.

[camilamacedo86]

Hi @pohly,

Could you please let us know?

a) Have we enabled the metrics in your project? if yes, have you installed Prometheus on the cluster?
b) Could you provide the project/link for us to try to reproduce your scenario?
c) Could you please try to get the details of the deployment for we check if there has a reason for the failure as well?

[pohly]

a) Have we enabled the metrics in your project? if yes, have you installed Prometheus on the cluster?

The operator supports metrics collection and has a "metrics" port - see the manual deployment YAML. Is that what you mean with "enabled the metrics"?

Prometheus is not installed.

b) Could you provide the project/link for us to try to reproduce your scenario?

The project is https://github.com/intel/pmem-csi. First bring up some Kubernetes cluster without OLM installed and set KUBECONFIG.

Then run:

git clone https://github.com/intel/pmem-csi.git
cd pmem-csi
git checkout v1.0.1
sed -i -e 's/OLM_VERSION=v0.18.3/OLM_VERSION=v0.19.1/' test/start-stop-olm.sh
make _work/bin/operator-sdk-1.6.1 operator-generate-bundle
test/start-stop-olm.sh start # you can ignore errors about `_work/pmem-govm/ssh.0`, that's just for diagnostics
kubectl create ns pmem-csi
TEST_BUILD_PMEM_REGISTRY=localhost:5001 TEST_PMEM_REGISTRY=172.17.42.1:5001 TEST_LOCAL_REGISTRY=172.17.42.1:5001 TEST_LOCAL_REGISTRY_SKIP_TLS=true test/start-operator.sh -olm

This needs a Docker registry. You can use some external one like quay.io. In this example, I am running one on port 5001 of my build machine, which can be reached via 172.17.42.1 from inside the cluster. It doesn't use TLS.

Once test/start-operator.sh fails, you need to clean up before trying again:

test/stop-operator.sh -olm

c) Could you please try to get the details of the deployment for we check if there has a reason for the failure as well?

Which details do you need?

[pohly]

Also can you please mention the logs from the installplan if it exists in the cluster.

This is the installplans.operators.coreos.com CRD, right? There is no object of that kind after the failure and also none while operator-sdk is running.

[camilamacedo86]

Hi @pohly,

Following are some comments inline:

The operator supports metrics collection and has a "metrics" port - see the manual deployment YAML. Is that what you mean with "enabled the metrics"?

You enable the metrics in the config/default/kustomization https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/config/default/kustomization.yaml#L24-L25 with its layout. However, by looking at your project you deviated from the proposed layout. Also, please be aware of: https://sdk.operatorframework.io/docs/faqs/#can-i-customize-the-projects-initialized-with-operator-sdk

Prometheus is not installed.

If the Operator is integrated with OLM and the bundle has a PodMonitor or a ServiceMonitor (which will be the case when you enable to metrics) the complete InstallPlan will fail on a cluster, which does not have these CRD/the Prometheus operator installed. In this case, you might want to ensure the dependency requirement with OLM dependency or make clear its requirement for the Operator consumers.

If your bundle are you shipping the ServiceMonitor such as https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/bundle/manifests/memcached-operator-controller-manager-metrics-monitor_monitoring.coreos.com_v1_servicemonitor.yaml#L1-L2 then the installPlan would fail.

That would happen with any CRD/API required for the operator works that do not exist on the cluster.

Could you please provide the bundle that are you using + ensure that the Operator image is published in a public space so we are able to easily reproduce the issue by running operator-sdk run bundle with?

[pohly]

This project existed before we started adding an operator. That determined the layout. We understand that this not how the SDK is normally meant to be used, but rewriting the entire project wasn't ideal either. Thanks for any assistance that you can provide here despite the unusual approach.

Looking at what you said about enabling metrics my conclusion is that we don't enable those.

Here's the bundle content:

pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi.intel.com_pmemcsideployments.yaml 
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.3.0
  creationTimestamp: null
  name: pmemcsideployments.pmem-csi.intel.com
spec:
  group: pmem-csi.intel.com
  names:
    kind: PmemCSIDeployment
    listKind: PmemCSIDeploymentList
    plural: pmemcsideployments
    shortNames:
    - pcd
    singular: pmemcsideployment
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - jsonPath: .spec.deviceMode
      name: DeviceMode
      type: string
    - jsonPath: .spec.nodeSelector
      name: NodeSelector
      type: string
    - jsonPath: .spec.image
      name: Image
      type: string
    - jsonPath: .status.phase
      name: Status
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: PmemCSIDeployment is the Schema for the deployments API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: DeploymentSpec defines the desired state of Deployment
            properties:
              controllReplicas:
                description: ControllerReplicas determines how many copys of the controller Pod run concurrently. Zero (= unset) selects the builtin default, which is currently 1.
                minimum: 0
                type: integer
              controllerDriverResources:
                description: ControllerDriverResources Compute resources required by central driver container
                properties:
                  limits:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                  requests:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                type: object
              controllerTLSSecret:
                description: ControllerTLSSecret is the name of a secret which contains ca.crt, tls.crt and tls.key data for the scheduler extender and pod mutation webhook. A controller is started if (and only if) this secret is specified. The special string "-openshift-" enables the usage of https://docs.openshift.com/container-platform/4.6/security/certificates/service-serving-certificate.html to create certificates.
                type: string
              deviceMode:
                description: DeviceMode to use to manage PMEM devices.
                enum:
                - lvm
                - direct
                type: string
              image:
                description: PMEM-CSI driver container image
                type: string
              imagePullPolicy:
                description: PullPolicy image pull policy one of Always, Never, IfNotPresent
                type: string
              kubeletDir:
                description: KubeletDir kubelet's root directory path
                type: string
              labels:
                additionalProperties:
                  type: string
                description: Labels contains additional labels for all objects created by the operator.
                type: object
              logFormat:
                description: LogFormat
                enum:
                - text
                - json
                type: string
              logLevel:
                description: LogLevel number for the log verbosity
                type: integer
              maxUnavailable:
                anyOf:
                - type: integer
                - type: string
                description: DaemonSets use the default RollingUpdate strategy with at most 1 node not having a running driver pod. That limit can be increased with this setting, either with a higher integer or a percentage.
                x-kubernetes-int-or-string: true
              mutatePods:
                description: MutatePod defines how a mutating pod webhook is configured if a controller is started. The field is ignored if the controller is not enabled. The default is "Try".
                enum:
                - Always
                - Try
                - Never
                type: string
              nodeDriverResources:
                description: NodeDriverResources Compute resources required by driver container running on worker nodes
                properties:
                  limits:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                  requests:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                type: object
              nodeRegistrarImage:
                description: NodeRegistrarImage CSI node driver registrar sidecar image
                type: string
              nodeRegistrarResources:
                description: NodeRegistrarResources Compute resources required by node registrar sidecar container
                properties:
                  limits:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                  requests:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                type: object
              nodeSelector:
                additionalProperties:
                  type: string
                description: NodeSelector node labels to use for selection of driver node
                type: object
              pmemPercentage:
                description: PMEMPercentage represents the percentage of space to be used by the driver in each PMEM region on every node. Unset (= zero) selects the default of 100%. This is only valid for driver in LVM mode.
                maximum: 100
                minimum: 0
                type: integer
              provisionerImage:
                description: ProvisionerImage CSI provisioner sidecar image
                type: string
              provisionerResources:
                description: ProvisionerResources Compute resources required by provisioner sidecar container
                properties:
                  limits:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                  requests:
                    additionalProperties:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                    type: object
                type: object
              schedulerNodePort:
                description: SchedulerNodePort, if non-zero, ensures that the "scheduler" service is created as a NodeService with that fixed port number. Otherwise that service is created as a cluster service. The number must be from the range reserved by Kubernetes for node ports. This is useful if the kube-scheduler cannot reach the scheduler extender via a cluster service.
                format: int32
                type: integer
            type: object
          status:
            description: DeploymentStatus defines the observed state of Deployment
            properties:
              conditions:
                description: Conditions
                items:
                  properties:
                    lastUpdateTime:
                      description: Last time the condition was probed.
                      format: date-time
                      nullable: true
                      type: string
                    reason:
                      description: Message human readable text that explain why this condition is in this state
                      type: string
                    status:
                      description: Status of the condition, one of True, False, Unknown.
                      type: string
                    type:
                      description: Type of condition.
                      type: string
                  required:
                  - status
                  - type
                  type: object
                type: array
              driverComponents:
                items:
                  properties:
                    component:
                      description: 'DriverComponent represents type of the driver: controller or node'
                      type: string
                    lastUpdated:
                      description: LastUpdated time of the driver status
                      format: date-time
                      nullable: true
                      type: string
                    reason:
                      description: Reason represents the human readable text that explains why the driver is in this state.
                      type: string
                    status:
                      description: Status represents the state of the component; one of `Ready` or `NotReady`. Component becomes `Ready` if all the instances(Pods) of the driver component are in running state. Otherwise, `NotReady`.
                      type: string
                  required:
                  - component
                  - reason
                  - status
                  type: object
                type: array
              lastUpdated:
                description: LastUpdated time of the deployment status
                format: date-time
                nullable: true
                type: string
              phase:
                description: Phase indicates the state of the deployment
                type: string
              reason:
                type: string
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator.clusterserviceversion.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: |-
      [
        {
          "apiVersion": "pmem-csi.intel.com/v1beta1",
          "kind": "PmemCSIDeployment",
          "metadata": {
            "name": "pmem-csi.intel.com"
          },
          "spec": {
            "deviceMode": "lvm",
            "nodeSelector": {
              "feature.node.kubernetes.io/memory-nv.dax": "true"
            }
          }
        }
      ]
    capabilities: Seamless Upgrades
    categories: Storage
    certified: "false"
    containerImage: intel/pmem-csi-driver:v1.0.1
    createdAt: 2021-11-22T08:19:12Z
    description: An operator for deploying and managing the PMEM-CSI driver
    operators.operatorframework.io/builder: operator-sdk-v1.6.1+git
    operators.operatorframework.io/project_layout: ""
    repository: https://github.com/intel/pmem-csi/tree/release-1.0
    support: Intel
  name: pmem-csi-operator.v1.0.1
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
    - description: Represents a PMEM-CSI driver deployment in the cluster.
      displayName: PmemCSIDeployment
      kind: PmemCSIDeployment
      name: pmemcsideployments.pmem-csi.intel.com
      resources:
      - kind: StatefulSet
        name: ""
        replaces: pmem-csi-operator.v1.0.0
        version: apps/v1
      - kind: Deployment
        name: ""
        replaces: pmem-csi-operator.v1.0.0
        version: apps/v1
      - kind: DaemonSet
        name: ""
        replaces: pmem-csi-operator.v1.0.0
        version: apps/v1
      - kind: Secret
        name: ""
        replaces: pmem-csi-operator.v1.0.0
        version: v1
      - kind: Service
        name: ""
        replaces: pmem-csi-operator.v1.0.0
        version: v1
      replaces: pmem-csi-operator.v1.0.0
      version: v1beta1
  description: |2

    Intel PMEM-CSI is a [CSI](https://github.com/container-storage-interface/spec)
    storage driver for container orchestrators like Kubernetes. It makes local
    persistent memory ([PMEM](https://pmem.io/)) available as a filesystem volume to
    container applications. This is the operator to deploy and manage the
    [PMEM-CSI](https://intel.github.io/pmem-csi/1.0/README.html)
    driver on a Kubernetes cluster.

    Refer to the [PMEM-CSI documentation](https://intel.github.io/pmem-csi/1.0/README.html)
    for more details on deploying and using the PMEM-CSI driver.
  displayName: Operator for PMEM-CSI driver
  icon:
  - base64data: iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYAAABw4pVUAAAx6ElEQVR42s2dCXhcdbn/BxQRRVkVEOSK+4JXBFRUqrKI3CsI/FEUEBAURFAg3ehCCy1tKYVCm+4tdKH73qZrmrTZZ18zS5LJMpNJJslkstL73PuI9/H7f9/f75wzZ07OTCZt8f/v8/yeLE0mc36f8+7v7z0WmPz75z//if/93//Fyf/6L6R6e9He0YHWtrhYLa0xsZpb2rQVbW5FU7RFW41NzWI1NEbF4s/5+/xz/PP8OrF4AvH2DrQnOtHR2YXOZDeSXT3o6k6hJ9Ur/m5vXxrp/j70DwxgcHAQw8PDeP/9909pGX+Xv+bXHKDX7u/vRzqdRi/9zVQqhe7ubnR1dSGZTKKDrj2R6BDvty3Wru0BX4t6nZGGJoTCDQiGIqgPhRGoD8Hnr4fXF4DH64fb44Ofvo7SHqTTffj7Bx+IPTb7ZzF+g0EM0xvuTHbRm2hHW5xAxOI5gfAb0wPRg+A3agaDL0yFkehIZmD09KCnN4VUWgdjcAADCowzBYQ/HxoaMoXRQ+9BhdHZmSQYfOOMhKG/1nCkUcIIEoxgEL5ANgynywOH0w2b3Qm7w0U/F0J3TwofEJi8QPgH+M7kzVLvCP0bKQTGaJLBrxdrJyB0oQm6+1g6MpJhAmNoEIPDQ6cFwwjGCIOXGYwOulniLMnKXuSCoUqHkAwBwz8CBoNQF3/N/9dMr/Xf//3f5kAYRrKrOwuECqNQydDD4GUKQ1FViY5OIYWdXUkkSUV0p3qyYShqijfvTMPg12UYfX19JBkMo5dgpAhGD8HoJhhd4v3pb0x+/3wduWCwdPgDQaGqzGCo0sFf8/f5/3lFGhqzoFhUNdVDksF/PJ9kGIGYqSm9dKgw+PfaYnGhAln8WU1pdqO7S8BQVRXD6BvoF2rqw4LB0sEwVFUlYXRrMFgy2k8RBquq0WC43F7xMwyOf76tLaapLwsbl+Hh97W7Qf8mzoSq4t+TRrxdkwy9EdfDYOlgGKyqVBgfhqrSw5BGnCWjSwDJBUOvqrKMuAEGbzJvOG+8HgYvIwxe/Hv1pOpYUpmFhaUj2aUY8ByqaixqSoPRwjBYVZF0xA1GnPU0qyk24mw36M2kaZP6SI0IVTU0eMZA5PeoegmGlA5VMhIJqarywdAb8Vww9EacP+dllAz+Pf59tj38+h988A9YTp78L3k3xEe3GepdrwJRAZhLRrbdUGEII25QU0IyaKMGdHbjTMLI5VFJGBm7we9xNMlQVdVoMPQGXA9DBaLCUIEwYH6fFr47jXbDCESVjFzSwW9Ub8RVNZWB0ZFlN7p6dEY8LWGwmvpXGHHVo5Kxhs6I6yRD9aj0dkMPQ/WoeDNzScZodkMvHQyXX5PVuYXvCpaOXLHGaDCM0mEOI2M32L1VpUN4VH3SozodI57qH0Kkox81Tb04VN+N7a5OrK1LYEVVHMsqY1hW0YalJ9qw5Hgr1lS1YVNdG/a7YqgOtiPc2ilgSLvRLpyPzLXnh6F6VEbJOBUY/Nq8fxZ5R8RNbYYZEBWGUVWZubd6NaXB6JFReCqd1jwqNd4Yfn8kjGHD57z6h4bhaevDOmsCE/ZG8eu1Ify0OIBvz/fi6lkuXDzdjo9PsuLsojqcVWSlVaetT0y24dLpDlwz24nr33DjzmU+/G1rCGsryVBH+WYir5DsX1Mz33xNprGG0aPiDT9dGPw3GLwAokrHqcLg/1PdW30kbgajJ5VWYEgDLlQVBX5DBhjDJ8nDev+k8rkE0dDZjzlH2vCjRT5cPtOBT0+x42MTbfjohDp87mUnfrrYj4ffa8AkgjTvaAtWVcexproNq0kq3iprwaySJvxxYxC3L/Hiypl2nD1eAjuHfv/CKTZ8cbYD03YF4QjydfH1kXTQRoUj+WHoPSpeY4XBS1X7lraYuXt7SjBibSLVEqdInFVVlnurwujty8QaA1Iyhti9PTlSMvoHh9HSPYAd7i7csbye7m47Lpxqw1Wv0OYvCWDK/hbs83ehtXtQwBs+eVJI2SCpPobd1z8o7EZKiTWEe8selchRdcIaimHuwQhuftuFS6YzWKsA9MVZdhQfCcITbFCkwzzWyGU39DB46WHwygWD99NyxmCY5KgyaZEemTBME4y+/mw1pc8xKR/TBKI22ovZR1rxg7ekGrptaQB/29mELc4OREhSBoez1ZmZRyUj8V4lEpcuLmcHEp30HjsSiCdItdJ7bqRr2GVtxKPrfLjmFYdQbZ+cbMXj6zw47gyK3FOgPigShKPBMEbiY4GRBSRX9tYMRi67YQ4jJRJpmhHX5ahGqCm6y+vb+zB+T5T0uw/jSAXNPNiKEpKCpmQ/SVK2LRkNRiZHlXFxpXvbITIGwoiL6+frbkWYVNTGijB+XuwWauwjtO4oduKYjTe0Hr4cCcN8doNhqEBUGEa7od9Py1hS6cZfzqRF8sGgu5NcaxWGUFUKDKOaqmxM4d9IGn6+PICd7iSpIlI7QxlbMqTaFB3A0WH0CFWlwhAJQyUSF45Ma1zAkB4V3WwNjTjuiuAXBOIjZGPOITX24GqyD26SDIZBm60acaOa0sNgydBLB8PQSwfvpX4/1ZDBoqqqXO6teY6KVFUrxRttmdoGw1BrG6qaEpKR6s2CoaVFDHf6sAJkpycp1FFeV1cBOaT8HjsFA4YcVUYyZKzBN4qqqlT3NtTUhvrGFtQ30HUq1+oLNaI+1IByRwhfJltiIfV12XQrlh6k4M4TOKMelVlG3GK0G/lUlVEyjIUmtbZhpqZUGGoqXYI4iffsnbQ6UNXYQ5IzbKqOzFxh6XmRZCgw+gcGtUhcD0OmRZIiFmpPZGINTun8eaMfD6z2YsJ2P6q8DaihNW2nF+vKSDX5Q1h80IuPkpRYxlvxwEoHqu1kN5yn51EZYej3VADJp6r0hSY1Epcw2oSrHIvnKDSpVb+0kqPSwRgSmzgs7AFv7Pnkbp7/og1Pbm1Ed9+Q2OTRgZwUP3OYgsDNBLPE24lYV3pEwpBrG7KuoRhxrdAUE9dwFbm+bMC/MdeBPbVhrC2vxy1vO/HsRg9qXPVweQO4dq5N/MwXX7HiSI27IBi8jDB46WHoMxt6W2wpJArPhpGdvc2C0d2lpEVSWglW5qhInZCaYggDBIPV0tFQt1A5ImAbX4vHNkUIiM6ukAurX8OazTgpVBaDu508L3aDf/iWD1WRpEhScrKSk5Zd3d3So+LsrTDi0qNqbW1TnJhWXPWyAmSOA7trQnAHIlh4wI8tx2lj/bLQ9MwGl/iZj5CUbC8jGAoQFYYKRC8ZvMxg6IGo+23UOJZC7EbBJVi2G0r2NtuIDwmpGKS135vEtfM92O3tEpu8gVTWBptUWf2KAR9W7YSy1MBQjTOEESdpu+ltn1An185340SwQ4GREpnkpHBvuyguSqC5TantxOiidTmqDBA7dlUHEWTDK4I/vss5LeLDwv0uLcpfecAhgIxmxFUYhRhxY9hgyRVrGBOGeo8qU/XLVlOaZKTVql9/pgRLG8p5pqtnOYVb29Unv7eiuh0rqtpxLMxAZF5qs6NTfH9ljVz8+arqBAWIncLz4sBvgFTgTSQZHMhdRVH6i3sasbi8WdwU/J780QTWVzVjxr4wnt8WxJySMHZbm4RDElWuVwXydQKys6o+E4mr7i1t9tIDKhArlpSceSOenfsjIKOpqVww9N0i+hxVpgTLVb8haTfojvbG0rjuDS++T5sYpZhCtQfq3ffYpgZ0EYzmrgF8ba4nK/8kVUYdRdN2/HZ9CNFOmZDkoNH4cxyJW8PteGhtAJdMk14SbyanV74824H5B4KKR9WEK2faNCA7KqXe9/ozsYaLNn3iZof4fX6d9UedZNTPPIyM1klIIGptY7RYw7R1R9Q2Ulkw1BKs8JoISO/g+3h6eyM+OtGKPaSy9AZa2hAJpJuBdGeAXDzNge8t9FGQ6MFF0+xiYz41xY7llW3i76hAOKr+xjwXrl/gRnO8E89sDeKciTKpeDVJwY/fcuHTU2Ra5AqCsLkyJGrZV+mAbK/wZ3WKCPeWNn7cG/J1PvWiFfsrCIjLe9ow9CUKvQngZTGzG8a0iKyH5+ijIiNuLMEKNaWkRdgmHCMD/uU5Lty8ODDCY8oFhO/Ie9aEFPtxEsuq4uSNWUW+6W87GkR/0w8Wyp/71mtulPrjIkdlI+n4zEtSFd21wotoi/SoXj8YxPmT5e8/ttYjNuzKGRkg2074s2obrJZ2H3eJiP2solrc9LoV5XUeITX5PCpeehjGVigzGPq9teQsweoLTUpzQrto25HdIqwaunpSWgm2V5Rg+4RuH1SMONsIjrRfPtQqpGM9Ge/hAoHw9+9lIBwA0muxe8seFUfOz2yLiBvgBwu9BMRKQFwo9cVEwnBjdbOm81/cGcTaExG8S2vBgXrx+/x/ty0mt9UXJiCKDXnVhm3HfQoMn4g1TtS58ItFUlV9jN77Cxvpd5zegmGYpUXM7Aa3GLVrDhJJSC4YZrUNvZrSJINhpHX1cLXQpARwsdQgfrYkgM/SxYcSfSPSJTmBjJdA2FVmwJvsCdpQOwGxCSD8txkIb/y3SF2VemPigl49EFHsBtmcaVZ8ZrpNrEum2YQd4p+/eaETFc6QIiFWfI2AbC33ajmqOrsbEzc58MlJUjq+/qoVW8rcCowMEBWGCsQIQ7XNuY24hMEFMl4cM1lG6zDUe1T6EuyIDkOTPirOPXFm9kLS/+MW+RHrGShcQgSQIL2ebFDYaGvXgPxla1hUHfVAjnraRKyx8EhYbCJDuW+VB4+t85KKkutRWo+868LMnT5YPQxE2pWvzbbRhnuUuoYH83Y58JlpEh4XuiZsorjDRSAEkIxknI4R1/ZVgyGDWIuZeys6RXLAUFt3smGM7KNSE3+VDb1ig+5ZHUJH76AM7EYB8lUdkMFBmTTcaI1rKouBsJTepAD5xlwnDrlahNu4g1xbDuL4dV8rCYoik1r1s3oicPnVWCOoGHUJZPMxN6qtHkzZ4hBOgvTManHHIgesTmk3vATDd0bc24RmxBkGJzwZCMdOlmwj3qrAiOeGoeSo1Ka2fqVbxKwezlB2ezrF5vxuHW1iekgDpa6PT7KJu/mxjTJSb+4aJCBu8Tu/IpUlCk19/XivLq5ISB2e3hISrUu/XOET4FgdPUexxrKyBjhDLSQxTvH7n6MNn7u/HmvKglhQEsCP3nTiyfUURNplNP2tudKGMJBNpW68vN2pVBHZc6MbYpkdVbZT96iM2Q2j3ciG0S3ybhbzpjZzjypTgtV5VHlad1g97aGIXBjSpQHEU4MjVNbtywICyKMbGySQZMao/35DWJR62aPaUJuRkKc3h4RHtaYiKjaQDTu7uZ+gO7sh2op5JSFcPNUmvn8uff+zZEPUu/5Ccn+XHvKJtEjRJpcCxIqNFGMUvScBXTG9Ds9vdAivyu3xjgmGPi2iwshlN1QYvLg8wPk3SxaMWEwpwSZyw+jty+owzHdMgDe/tiklNvwLs11oos3mWrn+ZzyxPmH0n9jSKIBEyeaoEvL01oiAwV1962tjuEAB8ufNQSTI22uli1t4tAFfedUh3VNSVdHmNtQ3RLH0aJAkxaF5XBbh3trwZokHNpdf1DZq7D5M3uzEvUvt2HrMSZ/bcfcSticu1DkzMDw+8z4qterHQPTZWzMYaldo3AQGSwcX0AQQ8z4qY7dIT1Znur4Em691h3NPkc4BMurS3dytBIVail3NUZEa6x0cFgZcFprM+qgyTW0dSlObvtBk1tQWiURQ6w2jzBFCtStHU5tDaWqzZxrbRmtqy+fejmY32kfASIqsNPeJcTHNkg9GJkelpEVU97agYwLSVsQVt5fVB5dkk31DWVC0lbeprScnDLNjArKPSt69nDAUrqkuR5VVDycI1VZyCqpcsDpGT4vkK8EWZMQNkqGHwddq4aofq6mY0ili3rqT6aPqH9DBODl6IamP7vxXODAkVcNxwNSSZgoWJTA9ELXqp4chO9NlkUktwSZMYKjSMbKpLZzduiPSItlNbTvLHPjju3b84R0HKqzuMwZDTTcZvdVElhHvETBUIHy9lkwknq+Pqs+0BFtoZ2FlQwrffM0jdPkF5CnNL20TmV2uc4jAb1hp3ekfGAGjW6TS+SI6leNliREw9OkfY2d6IKAcLyOXVauHE5Dt5U48sMKGS6eS4aeb5Wdv2XHC6jntqp8ehmo3RnpUGTWllw62lRZ9bSMbRmrEmQ1hxE+hM51r5Jw+4a5BC8UkXCF8cR9dSJJfc0jEGv0mZzZGnvXji2sfIRlqcjSrM11RU36/bE6ocfhwtMaDRftduGWhDedNzO5qZCAVNu8pwcjnURnthl5NZcPoFddu0cPglekU6R0BQ21qO5XG5wQFhezanjNBbsLZ5BFxgWnR8RiqG7qR6OkT5V69ZGRgyN7b9hxGXA9D7Ux3+UI4VOvHmqNezNjhopjCgc/PtGlpFW2RGv34RK6ZOwW0XH1U+kKTEcZoRjyR5d52m8LQgGQb8S5x8DKrBKuTjKHh4TGpKqM9aesZxGSSDG4BVTfk3ElW/PvrHvzm3SDmHo5itzMOXzO9Jz7upsBIqH1UrJOVenizCqMpimCkCdUUhW+v4gDQj2c3unHXUidumG/HlTOt+NhE+bdGwKD17bk2TNjswt4Kn5AkYx+VseqXK+9nBkNNi3SY2A0Vhl46WENYsjwq5ZiA2i1idG/NjhcXDESpg/cMDIus7zUUl5w1PvtOZZV28TQ7rphhx1fncK+uD/et8uPPm4J4amM9ntwYwJPvBfCnDT489I4Xdy3z4IYFDtH6eflLNlxC9uBTkzMAVAiWIj2QWpKIOvz4DTsW73dT8Ec2xe0XgWK+prZCS7Ca3VBAqDBY2vWqygwGOzSWbPe2x7x154wcoJH18CElnR5JpPHsjgbaSAfOm2QTGyWSghqkWmFvznqhVvm/uhzLqiUTM69h1b7mvNYnJtXiIoL1jVcpyl/nwK7jTjjdHmHkZY7KLyqFp5KjMiZitdDBkKPSAzFTVRoQUdfo7jHto1JrG2fseBkZdw7+ZB/VgKifOKNdmH2wCXevCuDa19xCQs4en32HqxKkLksWjGxAXKr9zDSZMPwRScGvl9sxbYsd7x11oM6ur4f7TrsEq4cxWo6KYykObrnHWC8ZehichrLIs34pYcR7dWc2ciUMz+RZP7Uzndt2GuNJHPPHsL66BXMPNuDpzUHcs9KPnyxy47r5TlEPZ9X0xdl2XEMfv0JfXzffgR++6cCdxU48vNqF5ze5MGe3R3QZcv3iaDW37VC8cQY6DHP1UeljDXMYPZrdMFNTehi8NxbNiBub2v4FMIRHRVLJ74EdCtlHRZ4UGfDG1nbUN7XBE2mDM9QMa30Udb4oar1kwL0RVLnDqHaHxKpzB2H3yMY2j8mZDbsju6lN30dlbN3J19RWiBFX7QbDyBjx1Ag1ZQZDANHXNvqU42WDp+HejgkGHxFQzoiLIQLcCC0O7LeLYJWzCGpnelOU1EVTdMQp2Mz0BAlD60wXkiFzVLma2kbroxqLR9U+Inurh5HKaTP6DYddLVmtO/SfH/Yp2MyZDXWURfYpWP2RZLNI3AhDf/BShaFv9/xw+6iy1dRIu5HfgBthCCD5qn4fziiLXt0xgdzTE0ZLi5jBMB4TUI8k54Khr4efaglWDyRbTfWIazRG4vlg8H5Z/hWjLPLnqMYGQ1UtuWB8WMcE9JKh76PKZcRzwTCzG/p9t+jVVKEwhnUHMbUGaENpVhwTEKqqX6Ts2YMTzdBkN7qUURYdnUmtGXpk9lbpNVbTIlwMEqpKHi/zUdzgNYFhtzEMo2T4zbO3wUyhKSJgNIqORj64E43KczD6IQhcwIvF4uTeKjNblNYdbk6QrVGZyl9PAR6VXjrUfbOcinubXcs4OeJMhzTistDEMNLCvc10pie1MxsMI6EY8bh2TEDmqFTJIBARHYxAZjiY22uUDNcIj0qbK8KdIv6RxwRCWR4VwWhsFn9bwBDjQWSfgdprIA+0KufvtYSshMHXJcaFKEbcTFWZwdAaQ1hlmTYn5IHD6XLu011VkxCHMosr4uLkU99gtkeVTPXjWH0SB30dOOBNoMQTFyvQ3KHlqCItCey0NmNnXTO210axraYRW6sbsIXW5sowNtEKR7Ilg48KbD/hg9XpE5G23ohXW11YfdCJ7eX0tTM7lV7jDGBjuR9l9nrxWgxjW2U9Np4I0gphU0VY/M0tVRFsp79f7WvCbmsjjrijsIda4WuKYW+kE3vCSbh42AC9/8OeNux1tsHWkBBGvJbc8f2VHtTQRwZjFvgNGNql9DCEUS9UOlQpmHW4VRxL/tQUG86bZMX5L9px+QwHntnehGSa6xrDFMcMwtncg6/P5ePGDtEkzY3Pl0xz4AuzHJhV0oDmWAIHXC2iGeHiqVb5kSLsi7K+tokjycFQSKTTy+0B8X98iGZ9qVv02ernihysdIkUybVzbNhwzJ2ph/vqBYwvvGzDjJ1eUUVkNfXNuXbxdy5S1sVT7eL3+Qz7yrIGLD0WwdySEB5e68dRTyueqkvh4pJ+/KY6jSAFsm+VNonMwu1LfbA3duJIrQ+fuesl3PDEApRaA6KbU9jovmzJMALRe6KWQlUV1zSmH2gRlb+fFvtFfTycICkI9eCBdWGRGHxicwN6+weFx2Zt6sJlL9nx08VeHPS24wBJx+qKKL7/pgeffcmGdyqjCEXjdBdGsauuCWuPR3ApAfjaHDu2VtGdWhnE5oqgrupXj2c3ukSj3CfpRpi02SFKrnojzkC40YHTKw+ucpIUyaQhH2deV+YXm/3iNq+mpq4iQJ+dbsW26jB2VJNk1DRgZ20j9pBkeMItqKtvgS3YigBJRz2916Z4J75d2o9P7BvEHG8v2klV3bvSLxos/ro9jLbOHkxdvhcf+9kE3DdlDZpiHdJ29A/kNOLGOV6WQu0Gnxvn6Qk/eMsv5orobUhb9wAe3BAWPbAHfJ1kMwZgbZRA7l0dEKeZ2Iiz3agLx0Q295nN9WhobhOjLDjwc1AkfvkMG2543ZFVglWb2qppc786247vv27D7Yvs+OECG47VZHtUB6vcos5y0RSrOKi54rBHM+LrywISyHaf5lExEJYGNuKq3Whu1o2VIiPO+SnOHIiRhGQzDtON9qXDA7j60AD2NfXAHe0UTd9cUlhV2Sq67++Zshofv3Uypq0sISnJraqMMERysVAgc8henDfZLmyHUZVxVL/D1SFaRn+/PiSaIeoaOgmIA/es8itpEWk3vE1xfGe+U6iBYFOLdiTZXt8kUujXv+4UHlVQsRvqAZqF+z3iSMDb+1yYvs0pUu3rDjsUGDIKP1zlwUcJyO1vO3DTAjupNrs4K8gStr48A0Q14nxgh88ZirSIYsQznZvy2J4xLRKj9aY/hUsODOK7xwbgaU9hszVGqtiJz5MqLw10iBO833lsAS74xVS8e6BWhBSFwBD1kEKB/GFTRGRhg4m+7FhjWJZgw/TGriAJ4j5bjv5rIx24jCThntV+7RQsX1ipt5XuJhsmbA+Kpjbp3jbD5m8UQL5LEhIyDAdzkNv6f5Y7cfXLVlJTHrx31IWvzLLidyvt0qNyyuztoWqfODX7+zUuvL7PgwterMNT690C7IbyegFkyg6/FmswEO56LD4SQnFpGEtKG7C8LIJDrua8aZFoRzceqUvjnH1D+M+qPrSSqzttfyPZ1Dp89w0PgrEerNlXjU/fMRVfuH+2OOFrZsSNMETFsFAgD2+QXeV8DnCkeztEb7IXV9IF8hgMdm9rwx1CZfFRgZf2RTB9bxjPbQ2KTf/Kq3byqhqyjpdZfQ0aENG6o5srsv2EF98k28K1DAZQWefCnYvtuHBKLaqsfGbDK7pJDtd6xfSFR99xiUObdxRLo73pOBl1WheR0zCVgIjAjxarKwb4WXI65LIJB6WIbpZcJViRWSAVvKmhFxeVDOKyg4PY15ImjZAk2+jEBaQuV1a2IZ7swdcemkf2ZCKmLN9vCmPQpMumYCBPbWsUbTzO1rSmpoRnMDgojJanpVvYmO8u8AiXrzqUoDfoEHfg995043tvuISuvW+VF+sqImiMqn1UMvCrI1dRBSKMuALDRdIxa6dLtIL+6V07ZmyT6/a3ZRfjtK1Ocn99IvhjL4frIY+964afXOQj1iBJlQ2/XOpC8eF64aFN3RlQovBmoa4up5umlFzbY54oyjzNKPe2wN0QGzGZojOZicTt8R7ceqIfH9s7hMnufoTp64c3hIRxf5BUdkO8myDsw3lkR37yTDHCrR2mqkpNoehHDhYM5K3jMXzyRRvmHW3VzmwIGAMy8FtV1SZU0dNbwsInrwq2ExA7biv2oirQhgp/C31sRaipJXv2regwbEStRwEy36GoKpkWqbJ78Z90p7OR/ve5dfjOXCu+M8+G62idP5m+nmdHnVPGG0fq/ALIowRE9lGFMW+/nySV1Nsat7h7p+0KoEmZRiFtiENnxDOTUzmVo6kpMc5WZnDb6M5/zpHGeeRp/bZ2ANFkGgtKW0gdkqPxtg/1sV5sOerA5+97Bf92/yzU+qJa5lwvGdrIQTW/1yv73woGEuwYwJVktNh41UZT4kCnfMF++Fu7cdtSv2hsriQQbMQrgnHh3t690qukRZQhL0o6ItPuKT0qrmsIIK/ZhREXaRG3H/so0OLeqfHv2XGkyiWKTqU1bpTVunH/ClZbdVh12Cuk6ahVAnlsrVvELmGyRVZfGHeShPBxOJZwAUS07ShAaMkclYQhTzONzFHJymoKmyK9OJ9g3FhOWqEjjWoKFD8300nq2oFq8iwjrQlc94cF+OTtL2LXcY+wsbmMuJAMcoC4JsRq3htpLhwIq6k1Ne2i0Y3PC8453IISbwcWH2/F9xd6xWHMOYeatLRIBUkFq6y7V3jlKAtu92QvRte6I3pjFfeWe28ZyHUERD8C6fn3KJgkY/7OIUPCkGzGxmMe0dDwyDtOca681BbQAYmIoJIj/XfL6bVnyBO503fXS/dWGRzAwWcxGfPiY3ItKWvEiuNROCLtBhjdsMVSuPRAP644NIgtTX1o6kjhutfJ+yPJW1Tehg4C9sBLa3EO2Y3Jy/YilR7QJlaYwlBS8twLV1LhxjW/nj02ICx6G20J/MeKenx9nhtfnE3ezhwnflbsw8LSKGKJbm32LQ8H+/kSL57bFhilqa1BeEG17nrcSu7qb1Y4MvOoyHu6d6kNj6y2obyWYYycffvzxQ7yqpyodAZo1eNGskHTd3gVGNK99YYa8dwWH25c4MTio2EBg9/Tfyzz4Eaybd8Tyy0X2bufLPJgp601k07ns++02VNdfbitoh+z/f1IpNIUd7SJYxYTdjeihVRXmT2I259fhifnb6VgMp45SWaEIZKtMr8Xp71avO04vvibV2G5eXxhQFQYbDfYgDeRR3WsvhMlngQO+drhiWaSbOrsW56eUE02wxGii8/b1CZrGx66ww/W+ISnlMlRubGr3EHxhVLbMEmlHyM1ddRKxt/PQ/BDOGwNocYTzm5qo+UKNqHcLSNwcTCJgFQHWoRtq6T3yXauur4NNaE4akPtiMY6dfXwFJI9KYQ70wglKUJPpUX2uinRiwZa7d2yfBHrTFF03on2rpRmN9QEYpZ726fC6ELRol247K4ZOOvmojEAUWAMKLUNYYjS6cwoC26ETiY1GOqEz5a2WFZnulmHoZoWkSXY7GZozt7KDG5GTbndhkltXGgKhLSEYYjUXzjcOLIE20yrRVfbiMXlWRg24nF9o3lSXIts3ZEwukX2tlcxvunMNFXai341ChdHweVgHXXakR5G9kTttBi4efekVTj3lkkEg0CMGwOQXMPBZNNXakQJNqaMnc3dmZ55zka+Afb6YcRqc4LZpLbTOusXz3hUZiVYfWd6oVW/Ee6tAoOTjTzMrcodwbi/LMJZ48ZLCPo11np49qQ282MC+nq4CkM/blXACI6EYTap7cOsh+dr3VFBjKUEqweSNfhfVxPaVmrHjU8sxNm0+ZqaGgsQswM0+mborBJsu7lkGDvTJYx6LdZwm7bunDkYZke9zTsM9X1UPQX1UZkVmcxyVB0Uv7y24Qiu/j+zaOOLBIyzxo0BiD7Xwvn8kTPT1edsJKWqMpmZbgZDf2ZD1CsMMLKb2lQ15VXqGurxMn0fVVjzqBoEjEbZmR5Vpiew3ciaQx9XHijTISZTyKnbycx0iqyHBCjH+NK9o8IwAlEzGPw7re1JPPvmdnz6jikjARQCJFcflb51R5MMk9advDBoI120qVYvN7bJqh9X96rsHliVs36cQKxyeFFNEXi1KwC7JhkBCvTII9OMeBjOYCP8ZMTDoh4uD1/6G1vgbGyFv6lV8ahkJN5MK9yWQIRWUzwDI06ruZNuMKXHOdbVi46e3ozd6Mtd9TNTU2pjBxengtE4Hp39Hj760wliw03VVD4gufqoVH2qtu6M9miHXDC85J4utzXh4fJ2HHXUw0ZSsLC6AU+UteK4laSEYo336oL4bVk7Hi+P48njcSy1NgnJqPaGcH95BxZYabPp9U4EGvHgiSRWuWKaqtpD7uvjVV345YkUfl3Rg42BTi0tUhLpxEPVvXjamsIzNnLdm2SD3omWHjzvSMOT6EGQgr2JFG9UxNLZDYQGGPnS6f1KnMHjZTkuOVsYbxXEGIDklgz9I4FIMpJJ0W3BRZtcD8syg8F630XxRlFlDGfvGcTfKMovtftw3aFufKGkF4dtPjGPan5Vo/h6bnUzllkbsdcZIfc2iGO+CCz0ezcc6aX4pwl/qOjEx/cOYHxtp3Rx6e/eVNqLO4+nsNLbgYeqUvja4T5UNclJpMv9Xbi4ZACL/d346fE+PF7XK67HE+/GbSf6MN2Txnx/moK/Afg60yOMuFkJNltNDQjJ4EOy7+6vwbW/n69IRJHhYwFA8ncY9ma5twll9u1oHYbZsYbso3J6AyiqaMOVtOGX7u/D+BMtuII+v/5QF47Y5NCw+dVNuLwkjV+XJfBURRyH3DIdz0A+umcAnz/Qh0cqunDVwT5ccyiNCXWd4u/WRlpxwf5BFHs6RN7MStHyp+nr1YGkuHmW+5O4kIC86unBjcf68aQtLVs+6dqWB9P4/KEBfKt0AMvCfaLXeaxGnCejMoyFW8pw2d0zhGQI433zC8oaAxCz5zNlPKpUdoehyYNP8h281De1OSiwe+FEK+4uTeBGgnDx/l48fCyGmw8ncdTuF0Ber4kKIL9RgBx2yzODZf4GXLivH78nybhofz8eqezC9472CiD89+sibbigpB9LPFJNWaPtAsiaejnNemWgG+fuG8RXj/Tju8f6hGRwWoSbvdvp4w1l/bixbACJVF9BTW1GjypOdmjGqgP42C0TZbCnSAWvc2+djE/+/EUBZlQguRJf+kHEGY8q90MWTU/B1oeyYg22Gc+TVPzqaByra4KYdLwZS6ojEohDJhUZyL8dSGNmdSveqGvGFmdUvFY5AbloP6kcRwxzbQlUBlvxA1JRE61JrR7+w2O9+MXxXqzysb0glXWkDzXNncKjWlnfjSsODmB9uAfXkYQsre/NHOGjO/vemgGx0qM0tclIXIUxJOIMWyCKB2eu11STWOzWKsHfj//8Nv40b1MmIs+1cj8SqDdrXrrq3sZyPGRRgxEZCUOfvbWTR1VMAGacaEC1XXYY7rYFMLGiGVUumT7ZbI/gd+UJPHy8A7+n9bq1TXhU1vomkookjgRatA7DySQd73gTSnNCKw6EEvhTbQr3VqTxWzLgW4JdMvjrkA0KfyE1FUx0422CMcXdJzK0KWUwwsJgHy15kCiXZGgwlHRSmmzG4doAfvJsMXlSE7M2l+FcdOdUfPm3c/Dvj76OxdvKzGMP/dJ7BtlPvNSNsjBRU2ZGXEhGWN/uaf7kshpybasYBrd8utyw0//VuelnhHtbDzctmy8Imz8MOy13UKbSGbYnHEWoIdPuWU+ubSSqdIookXikNY5Aa4Jc3E4lRyUfndHe2am4t91op2trTcopFephpY7ePrH6lByVGYz3dYlWrgntq/TiC/fPwkd+MsGwudJe3Pq3Ymw7asUrq/fjBGmAs0aLQ/TN0GbHBKR7m9+jMjPixmfBZj8SKJOjcuoO0HiUeVTapDYO/LT+W4rCIzKDy9nbJiVhKGsbKozsqp/6NNEET7dOJrXaRrdyhI9VlZhNr0zFSyuNbf0mfVRDOveWYXD2d9G246JMK9SUIS/FkqAa8fPJdvxq8iqU1nrx0Ix3R0hS1tJ3pmc/frQrZ45q1MeP5oXx4eeojMcEsnNUqZxHknPmqIYzh1W5gTzcksDzb+/CJ26bbIAwHh+/ZRJu/sti3DtlNb7+4Dwt7mAw1/x6FhZtKcWlv5yebWsUcJf+8iVYRs4VyX4WLF9YvICE4ak8fvTDgqH2UWUPecmcZjImDFUg+QM/CYQLT/dNfUeUaI1B3vk/n4KiRTvx14Xb8fxb2/HewVpc+8hryoaPF+uzd79E6m28lCjNLS7CV8nOLNlaCsvIx4+qc0U6NRi5RlmcyceP6mHo51GpI5AKPUBjNsoiM49q5CnY0dxb1W6wmrKSJ/XNh18TaZCzTKLuG//4pmi4/tIDr+Jccn0fm72BoNRIGOOkJJx9c0aVqevbj87HrmNW+fwQ/TPE9UNejO7tmXj86KlIRiHDwfJNajObnlDQmY1hpdjEs7z6+rG51IFL9KpmnC4Von5OH59buA1vbjwqvKsvPTBbdFNaRtQ95GuwLbnpyYWooIBYnqeMwSLd2x5FOroEEGHEC3jiZSGPH7XrniZgfPyo2bjV0WCc2iiLnpxqKnfCUJauO3vSeG1DKT5378ua6lHtBXtWnMGVqXQJ5FNkwGes3IfJxbvFx7snrjQx3kW48BdT8cgra+Gpb1A8RAVIBob6QPfOnENexvrES6N0FPL4UePYPOOk7dGGvOhhZNRUKmfVL2cjNM9m6egWDQsX3Tkt466OU430eNxFm128vRzjyIhzh6Ja42A3eMLinbjtb8WmHtUVv3oJL63Yg0C4SQGhLgKiJgxZMrQnl7XL00zqBpwpGKczHEwPQz+PSp0SNHJSmzo9QZ5o0sPgZe5RKQ+aISD+xrjY6HOEvch4SpZxGc/obnJlF24qxeKt5Zj9zgFcLMAViRzWeeSBnT1ugiFQfAEX/8c0rNpVLhKhKgT9sqh2Y2yT2swfsvj/i3srpcO8BJv3eBm7tRTslTsjuOGJN5Wad5HOS+LPJ2jG/LxbJ2HfCRd+8cJybD5ch0NkL254fIEOWgYGOwJf/d0cHKl2yevgpkEDDA2I6lHlgpHrSHK+J16eCgzjPKpcY/PyTfjUl2DHAoOlom9gEOsP2fCNh18bkTL/CN3tNzzxhmgP1ewIrZkUgS/dcRxX3fcKrn/8DREEGqNxjlfumrAcVRSpZ6soEwnJPBIoYXgwr3mhKR8M4zPEzUZZ5HrI4mgPy9LPMMw95EUdf9STNT2hkLN+ia4Upq08gMsoTshOkReJTsR7Jq/G1GV7Rc+uPti7/g9vYN66wyIgPGtEEapIRPJ/fm0Tat1BA4CRMAQQfQk2H4yQkqPy13+4jx81PrvECCPfIGKz8Ue5YOgzt83tXXh09ka6kw09UspiW7K2pIakYza+/6eFZKyXaHEI91V9meIOtfaR8cSKxCmq19YeQLAharr5pkDiBY3NyyQMcz222ti6M5Y+qlOd1JYr1ii0j4phBJrbRZpDbOg4abSNRaRP3TEVCzaWYs2eSryzr0oUn3ido6uTSwgviI8cibMkbTpYLer4uaTBFIiwG3lKsIUnDP9FRjznXBFzuyEPXfbL0SGq3VDS5geq/bjhjwt1Bnu8ElWPx+W/mklGeK7WzHYlxSGbyHD/9c3tuPfF1Xh2wVaKOaYYclLSeP/oqbew42id6HjhJ4kWCkPaELrA1gJLsP8vm9rYbpgHfl15Zxhq0sEFOGHEh9Dd24fiHRW4Rm1w1tUvONj7wZNvYdqK/XhnfzVue26ptulfemAO1h+oxeNzNop+3EwuarwG7v6pq1Fa68lyRApdfI0Wzu20FOhR5WpqG+sTL/M9ZHFsk9oydkMPQ51HpVdTAgbFGT0EY+aag0qwZygWjXsBn7vnZdGgcNNTbwtJmLn6gK6oVCTiCDXlnkm7F4nAcFLxTgr2GsekovSLSwIWnlGSrwT7YcIwe3ZJIUY8mcydo1KlIwvGwKDSnd4jIm+RbTVtOCgSLurU5fvwCkHjbO06kohfkYfFIDL2IpNCYdvDwwLmvnsAjc2tOrd27ECYheUf//iHKOjk6qM6HRiFTmozez5TzEQyzCZD55pHxXZDjiocFOcxKt0NuKNoRd6+KHWzOaZYs7cKLy7dgx//5W08vWALvv3IfNp8nRc1Ttqbbz40D0u3laGRruNUIKg5LC4xMwvLP//5T1EhYwnJNxwsH4zRHj+qAjFKhvFhWcYnl+UeDtZVwHAwqabYeO8sd4nIe2SZ1SwLK8Fwmn3ToVrlIE1RJoGoqCmWjFv+Wox9xx1544qCoNA1s6PBLCygf0yGg8N8MESDggGGXjpGG5tnpqbGMsOw0EHEmqpSDPg7+2vJRZ0pNvOsm4tG6Rx8IZMmIWlgw32uGvAJ6ZAtPB8hT+rBGWth94bkBKO2ttMCwq1IzID/CSD873/+539EfTrfpLZ/5XCw9pyp9FTu4WAKDG5YSyR7xGgLi9KgNmqT86iSIxenRp6iyLs+0qRVUU9VTamq6u9//0DFkAGiQuGYhLsEvR8CDLOnwRlhmOWo9DCMEz717q3oNqfljrThsVc34dxbJ2le0OkBkevz971Mxr5ETKCQM7ROT02xZPz973/XI8gGwv8++OAfok+JT8eeDgzjU2gKkYzcTxNQx+b15i3BsmRUextEg/O5t0w2yS2d2mJ1d/3jC7C+pFobZiZWa+spwWCpYJuhqqm8QPgfG5cPPvhAnAvh3ifedD0Mp8tzypKhf+ZV/hJsUvOmOBIvZIbh3goPrr5/1og6xCmDUGD86M9v4SgFe6LNSHE8xhz0tSdEnMGuLYPgPTb7938BhWWtdubpXywAAAAASUVORK5CYII=
    mediatype: image/png
  install:
    spec:
      clusterPermissions:
      - rules:
        - apiGroups:
          - rbac.authorization.k8s.io
          resources:
          - clusterroles
          - clusterrolebindings
          verbs:
          - '*'
        - apiGroups:
          - storage.k8s.io
          resources:
          - csidrivers
          verbs:
          - '*'
        - apiGroups:
          - pmem-csi.intel.com
          resources:
          - pmemcsideployments
          - pmemcsideployments/status
          - pmemcsideployments/finalizers
          verbs:
          - '*'
        - apiGroups:
          - admissionregistration.k8s.io
          resources:
          - mutatingwebhookconfigurations
          verbs:
          - '*'
        serviceAccountName: pmem-csi-operator
      deployments:
      - name: pmem-csi-operator
        spec:
          replicas: 1
          selector:
            matchLabels:
              name: pmem-csi-operator
          strategy: {}
          template:
            metadata:
              labels:
                app: pmem-csi-operator
                name: pmem-csi-operator
                pmem-csi.intel.com/webhook: ignore
            spec:
              containers:
              - command:
                - /usr/local/bin/pmem-csi-operator
                - -metrics-addr=:8080
                - -v=3
                env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['olm.targetNamespaces']
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: OPERATOR_NAME
                  value: pmem-csi-operator
                - name: GODEBUG
                  value: x509ignoreCN=0
                image: intel/pmem-csi-driver:v1.0.1
                imagePullPolicy: IfNotPresent
                livenessProbe:
                  failureThreshold: 5
                  httpGet:
                    path: /metrics
                    port: metrics
                    scheme: HTTP
                  periodSeconds: 10
                  successThreshold: 1
                  timeoutSeconds: 5
                name: pmem-csi-operator
                ports:
                - containerPort: 8080
                  name: metrics
                resources: {}
                securityContext:
                  readOnlyRootFilesystem: true
                volumeMounts:
                - mountPath: /tmp
                  name: tmp
              serviceAccountName: pmem-csi-operator
              volumes:
              - emptyDir: {}
                name: tmp
      permissions:
      - rules:
        - apiGroups:
          - ""
          resources:
          - events
          verbs:
          - '*'
        - apiGroups:
          - ""
          resources:
          - configmaps
          - services
          - services/finalizers
          - serviceaccounts
          - endpoints
          - events
          - secrets
          - pods
          verbs:
          - '*'
        - apiGroups:
          - apps
          resources:
          - daemonsets
          - statefulsets
          - deployments
          verbs:
          - '*'
        - apiGroups:
          - rbac.authorization.k8s.io
          resources:
          - roles
          - rolebindings
          verbs:
          - '*'
        - apiGroups:
          - ""
          resources:
          - pods
          - secrets
          verbs:
          - get
        serviceAccountName: pmem-csi-operator
    strategy: deployment
  installModes:
  - supported: true
    type: OwnNamespace
  - supported: true
    type: SingleNamespace
  - supported: false
    type: MultiNamespace
  - supported: false
    type: AllNamespaces
  keywords:
  - PMEM-CSI
  - PMEM-CSI operator
  - Persistent Memory
  - PMEM
  - Intel
  links:
  - name: PMEM-CSI
    url: https://intel.github.io/pmem-csi/1.0
  maintainers:
  - email: olev.kartau@intel.com
    name: Olev Kartau
  - email: patrick.ohly@intel.com
    name: Patrick Ohly
  - email: amarnath.valluri@intel.com
    name: Amarnath Valluri
  maturity: alpha
  minKubeVersion: 1.18.0
  provider:
    name: Intel® Corporation
  replaces: pmem-csi-operator.v1.0.0
  version: 1.0.1

pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator-metrics_v1_service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  name: pmem-csi-operator-metrics
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    name: pmem-csi-operator
status:
  loadBalancer: {}

pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator_v1_serviceaccount.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: null
  name: pmem-csi-operator

pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/metadata/annotations.yaml 
annotations:
  # Core bundle annotations.
  operators.operatorframework.io.bundle.mediatype.v1: registry+v1
  operators.operatorframework.io.bundle.manifests.v1: manifests/
  operators.operatorframework.io.bundle.metadata.v1: metadata/
  operators.operatorframework.io.bundle.package.v1: pmem-csi-operator
  operators.operatorframework.io.bundle.channels.v1: alpha,stable
  operators.operatorframework.io.bundle.channel.default.v1: stable
  operators.operatorframework.io.metrics.mediatype.v1: metrics+v1
  operators.operatorframework.io.metrics.builder: operator-sdk-v1.6.1+git
  operators.operatorframework.io.metrics.project_layout: 

  # Annotations for testing.

I pushed the image to docker.io/pohly/pmem-csi-bundle:v1.0.1.

[camilamacedo86]

Hi @pohly,

Would possible to just add the zip/dir with the bundle content? Otherwise, we need to copy and paste and manually generate it to try to test and see if we can help you out.

[pohly]

Would possible to just add the zip/dir with the bundle content? Otherwise, we need to copy and paste and manually generate it to try to test and see if we can help you out.

Do you still need that when the generated image is available (see docker.io/pohly/pmem-csi-bundle:v1.0.1)?

But I can of course also attach the original bundle files: bundle-1.0.1.tar.gz

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale