olm-spec-descriptors test validates alm-examples with the incorrect CRD

[tthvo]

Bug Report

What did you do?

Recently, the Cryostat Operator added another version to its existing CRD (i.e. v1beta1 and v1beta2). This version include breaking changes.

In the alm-examples annotation, we include examples for both versions. We then ran the scorecard tests for the operator, including built-in tests and custom ones, but encountered unexpected test failures. See below.

What did you expect to see?

The olm-spec-descriptors test to succeed.

What did you see instead? Under which circumstances?

When the built-in olm scorecard tests are run, the following issues occur:

2024-06-06T19:59:23.9902749Z /home/runner/work/cryostat-operator/cryostat-operator/bin/operator-sdk scorecard -n cryostat-operator-scorecard -s cryostat-scorecard -w 20m ghcr.io/cryostatio/cryostat-operator-bundle:ci-ceb2476d79d0d9b751ffcd31829e5ac867415d37 --pod-security=restricted 
2024-06-06T20:04:21.0721592Z --------------------------------------------------------------------------------
2024-06-06T20:04:21.0727195Z Image:      quay.io/operator-framework/scorecard-test:v1.31.0
2024-06-06T20:04:21.0772557Z Entrypoint: [scorecard-test olm-spec-descriptors]
2024-06-06T20:04:21.0774565Z Labels:
2024-06-06T20:04:21.0775082Z 	"test":"olm-spec-descriptors-test"
2024-06-06T20:04:21.0848319Z 	"suite":"olm"
2024-06-06T20:04:21.0848964Z Results:
2024-06-06T20:04:21.0849776Z 	Name: olm-spec-descriptors
2024-06-06T20:04:21.0850559Z 	State: fail
2024-06-06T20:04:21.0850804Z 
2024-06-06T20:04:21.0851157Z 	Suggestions:
2024-06-06T20:04:21.0851770Z 		Add a spec descriptor for minimal
2024-06-06T20:04:21.0852632Z 	Errors:
2024-06-06T20:04:21.0853030Z 		minimal does not have a spec descriptor
2024-06-06T20:04:21.0878998Z 	Log:
2024-06-06T20:04:21.0879995Z 		Loaded ClusterServiceVersion: cryostat-operator.v3.0.0-dev
2024-06-06T20:04:21.0880864Z 		Loaded 2 Custom Resources from alm-examples
2024-06-06T20:04:21.0881160Z 
2024-06-06T20:04:21.0881165Z 
2024-06-06T20:04:21.0881414Z --------------------------------------------------------------------------------

Reference: cryostatio/cryostat-operator#865 (comment)

Environment

Operator type:

/language go

Kubernetes cluster type:

$ operator-sdk version

operator-sdk version: "v1.31.0", commit: "e67da35ef4fff3e471a208904b2a142b27ae32b1", kubernetes version: "1.26.0", go version: "go1.19.11", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.22.4 linux/amd64

$ kubectl version

Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Possible Solution

Internally, the scorecard will find the FIRST matching (by kind) owned CRD in the CSV to validate the CR in alm-example annotation.

operator-sdk/internal/scorecard/tests/olm.go

Lines 280 to 319 in 0d54bbd

	func checkOwnedCSVSpecDescriptors(cr unstructured.Unstructured, csv *operatorsv1alpha1.ClusterServiceVersion,
	r scapiv1alpha3.TestResult) scapiv1alpha3.TestResult {
	if cr.Object[specDescriptor] == nil {
	r.State = scapiv1alpha3.FailState
	return r
	}
	block := cr.Object[specDescriptor].(map[string]interface{})
	var crd *operatorsv1alpha1.CRDDescription
	for _, owned := range csv.Spec.CustomResourceDefinitions.Owned {
	if owned.Kind == cr.GetKind() {
	crd = &owned
	break
	}
	}
	if crd == nil {
	msg := fmt.Sprintf("Failed to find an owned CRD for CR %s with GVK %s", cr.GetName(), cr.GroupVersionKind().String())
	r.Errors = append(r.Errors, msg)
	r.State = scapiv1alpha3.FailState
	return r
	}
	for key := range block {
	for _, specDesc := range crd.SpecDescriptors {
	if specDesc.Path == key {
	delete(block, key)
	break
	}
	}
	}
	for key := range block {
	r.Errors = append(r.Errors, fmt.Sprintf("%s does not have a %s descriptor", key, specDescriptor))
	r.Suggestions = append(r.Suggestions, fmt.Sprintf("Add a %s descriptor for %s", specDescriptor, key))
	r.State = scapiv1alpha3.FailState
	}
	return r
	}

This means both versions in alm-examples are validated against the first CRD def. In this case, it is v1beta2, which does not have minimal spec descriptor.

The solution is to also include an additional condition for apiVersion when searching for the matching CRD in the CSV.

Additional context

cryostatio/cryostat-operator#865

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[tthvo]

/remove-lifecycle stale