CustomResourceDefinition is invalid at kubernetes 1.18

[f41gh7]

Bug Report

What did you do?

create a new operator and add it api into it:

operator-sdk new test-operator --repo=test
operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService

Add init containers to pkg/apis/v1alpha1/appservice_types.go

// AppServiceSpec defines the desired state of AppService
type AppServiceSpec struct {
 // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
 // Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file
 // Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
 InitContainers []v1.Container `json:"initContainers"`
}

generate new crd and apply it:

 operator-sdk generate crds
 kubectl apply -f deploy/crds/app.example.com_appservices_crd.yaml 

get error:

The CustomResourceDefinition "appservices.app.example.com" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property

What did you expect to see?

add CRD without an error

What did you see instead? Under which circumstances?
error:

The CustomResourceDefinition "appservices.app.example.com" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property

Environment

• operator-sdk version:

1.18.1

• go version:

1.13.4

• Kubernetes version information:

1.18.2

• Kubernetes cluster kind:

• Are you writing your operator in ansible, helm, or go?

Possible Solution

1. Add a default value to CRD
2. replace []Container description at CRD with type: Array, type Object

Additional context

This problem is related to new 1.18 validations:

kubernetes-sigs/kubebuilder#1544

protocol - must have default value TCP:

default: TCP

                    ports:
                      description: List of ports to expose from the container. Exposing
                        a port here gives the system additional information about
                        the network connections a container uses, but is primarily
                        informational. Not specifying a port here DOES NOT prevent
                        that port from being exposed. Any port which is listening
                        on the default "0.0.0.0" address inside a container will be
                        accessible from the network. Cannot be updated.
                      items:
                        description: ContainerPort represents a network port in a
                          single container.
                        properties:
                          containerPort:
                            description: Number of port to expose on the pod's IP
                              address. This must be a valid port number, 0 < x < 65536.
                            format: int32
                            type: integer
                          hostIP:
                            description: What host IP to bind the external port to.
                            type: string
                          hostPort:
                            description: Number of port to expose on the host. If
                              specified, this must be a valid port number, 0 < x <
                              65536. If HostNetwork is specified, this must match
                              ContainerPort. Most containers do not need this.
                            format: int32
                            type: integer
                          name:
                            description: If specified, this must be an IANA_SVC_NAME
                              and unique within the pod. Each named port in a pod
                              must have a unique name. Name for the port that can
                              be referred to by services.
                            type: string
                          protocol:
                            description: Protocol for port. Must be UDP, TCP, or SCTP.
                              Defaults to "TCP".
                            type: string
                        required:
                        - containerPort
                        type: object
                      type: array
                      x-kubernetes-list-map-keys:
                      - containerPort
                      - protocol
                      x-kubernetes-list-type: map

[tengqm]

wow, this looks like a sig-apimachinery issue.

[estroz]

Yeah this may be an upstream bug if kubebuilder is similarly affected.

/triage needs-information
/kind bug

[estroz]

/assign

[kensipe]

@estroz it took some hunting... but I found the open issue around this: kubernetes-sigs/controller-tools#444

There is good detail discussion on this thread: kubernetes/kubernetes#91395

the core of it is:

So, the only missing piece is that core types lack any way to indicate default values.

Indicating defaults in struct annotations has been on the backlog for a while (xref #25460). Note that not all of our defaults can be expressed declaratively (some defaults vary depending on values of other fields, for example), but as far as I'm aware the ones used as keys in list-map items are constant. A KEP proposing a specific approach would be welcome.

[f41gh7]

It seems like removing core Kubernetes types scheme from crd open API is only one working solution for now. Probably it can be solved with some kubebuilder api marker in the future.

Should i address this issue to kubebuilder?

[kensipe]

/cc @jennybuckley @jpbetz

[estroz]

Seems to be fixed by kubernetes-sigs/controller-tools#440 kubernetes-sigs/controller-tools#480. We’ll need to bump c-t in kubebuilder then bump kubebuilder for this to be fixed.

[a1046149428]

Is there any update for this issue?

[f41gh7]

I've decided to remove inherited open api scheme for kubernetes built-in objects from my crds.

My motivation - it makes crd heavier. So, you cannot use kubectl apply -f crd.yaml for crd spec with few v1.Container fields. And some integrations breaks as well (operator-hub tests for instance).

[gmtstephane]

Hello, still nothing new ?
It is impossible to add Any k8s resources in our CRD, so it make creating operator quiet useless if we cannot handle its managed ressources directly in his spec.
The workaround is to fetch the ressources every time in the reconcile loop.

I would define this as a major issue.
Thanks

[estroz]

Workaround for post-v1.0 projects:

# Set CRD version so v1beta1 versions are generated
sed -i 's/\(CRD_OPTIONS ?= "crd:\)/\1crdVersions={v1beta1},/g' Makefile

# If you have any webhooks:
sed -i 's/\(+kubebuilder:webhook:\)/\1webhookVersions={v1beta1},admissionReviewVersions={v1beta1},/g' api/<version>/<kind>_webhook.go

# Then install the controller-gen binary containing the fix
rm -f $(command -v controller-gen)
mkdir tmp && pushd tmp && go mod init tmp
go get sigs.k8s.io/controller-tools/cmd/controller-gen@6fa696de4772fb44595d25355946254885a202df
popd tmp && rm -rf tmp
make manifests

[gmtstephane]

Hi, i tried your fix, without results, with a simple example :

// EcpmSpec defines the desired state of Ecpm
type EcpmSpec struct {	
	Foo     string            `json:"foo,omitempty"`
	MariaDB appsv1.Deployment `json:"mariaDB,omitempty"`
}
// EcpmStatus defines the observed state of Ecpm
type EcpmStatus struct {
}

Following your instructions, My makefile is like :
CRD_OPTIONS ?= "crd:crdVersions={v1beta1},crdVersions={v1beta1},trivialVersions=true"

and my controller-gen version :

➜  operator-demo controller-gen --version
Version: v0.4.1-0.20200901161351-4f077b9ee59f

When i try to install the crd :

Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
The CustomResourceDefinition "ecpms.*****.*****.*****" is invalid: 
* spec.validation.openAPIV3Schema.properties[spec].properties[mariaDB].properties[spec].properties[template].properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[mariaDB].properties[spec].properties[template].properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property

Thanks.

Edit : Same error if i use v1 instead of v1beta1.

[estroz]

@gmtstephane I linked the wrong commit in the instructions, try 6fa696de4772fb44595d25355946254885a202df (updated above).

[gmtstephane]

@gmtstephane I linked the wrong commit in the instructions, try 6fa696de4772fb44595d25355946254885a202df (updated above).
Working Thanks !

[hensur]

Workaround for 0.18.x projects:
I just bumped the controller-tools dependency and built a new binary.

The generated manifests have the default field set.

Required changes: hensur@ffa2f2c

[estroz]

Since this is fixed upstream but isn't available because go/v3-alpha isn't hasn't been added to operator-sdk init yet, so I'm bumping this to v1.3

[estroz]

This is still an upstream issue, see kubernetes-sigs/controller-tools#529.

I'm moving this to the backlog until this is resolved in controller-gen once and for all, and a patched v0.3 version is created for go/v2 projects.

[illeatmyhat]

FYI for those with recent projects, controller-gen 4.1 has a workaround

[jpbetz]

cc @apelisse for the map key defaulting aspect of this

[apelisse]

We fixed this in Kubernetes 1.19. Let me know if I can help.

[f41gh7]

For me it works, thanks!