run bundle - ConfigMap is invalid: []: Too long: must have at most 1048576 bytes

[disaster37]

Bug Report

What did you do?

I run

operator-sdk run bundle quay.io/webcenter/elasticsearch-operator-bundle:v0.0.1 -n default

What did you expect to see?

It deploy operator at the end

What did you see instead? Under which circumstances?

Failed to run bundle: create catalog: error creating registry pod: error building registry pod definition: configMap error: error updating ConfigMap: error creating ConfigMap: ConfigMap "elasticsearch-operator-catalog-configmap-partition-2" is invalid: []: Too long: must have at most 1048576 bytes 

Environment

Operator type:

language go

Kubernetes cluster type:

Rancher 2.6.8

$ operator-sdk version

operator-sdk version: "v1.26.1", commit: "4582a8414e65f40ebea65c74729f121e1f3e3b9a", kubernetes version: "1.25.0", go version: "go1.19.5", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.19.6 linux/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"2b525e8d2647a41e686bc7da5b7430667a13953e", GitTreeState:"clean", BuildDate:"2021-08-20T21:43:10Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.14", GitCommit:"3321ffc07d2f046afdf613796f9032f4460de093", GitTreeState:"clean", BuildDate:"2022-11-09T13:32:47Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Additional context

[varshaprasad96]

Usually the data stored in configMap cannot cannot exceed 1 MiB (ref: https://kubernetes.io/docs/concepts/configuration/configmap/#motivation). However, there have been issues created in SDK regarding the same previously. @everettraven @rashmigottipati are you aware of this and/or a possible workaround?

[everettraven]

Yeah, this should have been resolved by the partitioning logic that I implemented in #6182 however it looks like there may be a bug based on this error message.

It looks like it is correctly trying to create a partition but at some point the partition seems to have gotten to large to fit into the etcd limits.

[everettraven]

I'm wondering if the problem is either here:

operator-sdk/internal/olm/operator/registry/fbcindex/fbc_registry_pod.go

Lines 384 to 393 in d828db2

	// Create a new ConfigMap
	cm = f.makeBaseConfigMap()
	// Since adding this data would have made the previous
	// ConfigMap too large, add it to this new one.
	// No chunk of YAML from the bundle should cause
	// the ConfigMap size to exceed 1 MiB and if
	// somehow it does then there is a problem with the
	// YAML itself. We can't reasonably break it up smaller
	// since it is a single object.
	cm.Data[defaultConfigMapKey] = yamlDef

or here:

operator-sdk/internal/olm/operator/registry/fbcindex/fbc_registry_pod.go

Lines 400 to 405 in d828db2

	} else {
	// If there is no data in the ConfigMap
	// then this is the first pass. Since it is
	// the first pass go ahead and add the data.
	cm.Data[defaultConfigMapKey] = yamlDef
	}

That being said, I'm thinking the issue is more so that there is a single item that may be too large to fit into a ConfigMap on it's own as the partitioning logic is meant to split up the FBC YAML (which consists of multiple YAML definitions separated by ---) into single YAML definitions. My suspicion is that there is likely a single bundle definition in the FBC that is way to large to fit into a single ConfigMap. The partitioning logic in it's current state doesn't know how to handle a situation where it would need to split up a single definition into multiple (and I'm not sure how possible that is with ConfigMaps since this would essentially be a single file definition split amongst 2 ConfigMaps).

I would definitely be curious to see what exactly is trying to be placed into this ConfigMap making it too large to see if there is anything we can fix, but if it is indeed the scenario I mentioned above I think there would be a need for a much deeper discussion as to how we could go about fixing this problem.

[disaster37]

You can test it from this project: https://github.com/webcenter-fr/elasticsearch-operator

[everettraven]

Just wanted to follow up here and mention that I did take a look at this and the problem seems to be that a single olm.bundle specification is ~2.5Mb which is way over the traditional etcd limits. This definitely falls under the scenario I outlined in my previous comment where we can't really split up a singular definition.

That being said, I'm looking into what options may be available for us to take to remedy this.

[rnasani]

I hit the same issue today, is there any workaround or quick fix ?

[everettraven]

Unfortunately I haven't been able to come up with a workaround or a quick fix for this. I apologize for the inconvenience!

[rnasani]

I am hitting this issue only when more rbacs&crds are in the bundle(around 15 to 25 crds). Otherwise, it works without any problem.

[acornett21]

@everettraven I ran into this issue with a partner as well, I have a bundle that we could use to test this logic.

[everettraven]

Circling back around to this, the idea of using some form of compression has been brought up and is something we could do to help. Another idea is to add a new flag that allows users of operator-sdk run bundle to ignore the olm.bundle.object properties in the bundle resource (this field is so the packages installed with OLM are discoverable via the OLM packageserver) which, in some cases, could reduce the bundle size significantly.

Unfortunately, I don't think we will be able to entirely get rid of this problem but I think a combination of compression and the option to ignore/remove the olm.bundle.object properties from the bundle could make it much harder to hit the etcd limits.

I'm also not sure when we will be able to get the fix out for this - I'll take the milestone off so that this can be discussed in the community meeting next Monday (04/10/2023)

[abhijeet2096-confluent]

I hit the same issue today, is there any workaround or quick fix ?

FATA[0046] Failed to run bundle: create catalog: error creating registry pod: error building registry pod definition: configMap error: error updating ConfigMap: error creating ConfigMap: ConfigMap "confluent-for-kubernetes-catalog-configmap-partition-2" is invalid: []: Too long: must have at most 1048576 bytes