Enable compliance tests to use plugins for cluster provisioning

[tonifinger]

This PR introduces an interface that enables the provision of kubeconfig files for the sonobuoy test framework.
Each Kubernetes provider must derive its own specific plugin from this interface class in order to provide a cluster for testing

[mbuechse]

@tonifinger Can you please post what a run would look like in the shell? (Just paste something from your terminal.)

[mbuechse]

I think it would be good to use the test infrastructure that we already have, i.e., the scripts

• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compliance-check.py
• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-test-runner.py

and the spec files

• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-kaas.yaml (relevant for KaaS)
• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-iaas.yaml

The spec file for KaaS already includes the CNCF conformance test. What I think we could do:

• We make a list of clusters that we need for testing purposes, and for each cluster, we say where we expect the Kubeconfig to sit,
• then your tool can use the plugin to create all these clusters and the corresponding Kubeconfig files at the specified locations,
• it could also use the plugin and this specification to remove the clusters.
• We add calls to your tool to our Zuul job BEFORE and AFTER it runs the actual tests to create and remove clusters, respectively.

[tonifinger]

@tonifinger Can you please post what a run would look like in the shell? (Just paste something from your terminal.)

This is the output to the shell using the plugin of kind and the logging lvl set to info:

No kind clusters found.
INFO:root:Creating cluster scs-cluster..
Creating cluster "scs-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-scs-cluster"
You can now use your cluster with:

kubectl cluster-info --context kind-scs-cluster --kubeconfig .pytest-kind/scs-cluster/kubeconfig

Have a nice day! 👋
INFO:interface:check kubeconfig
INFO:interface:kubeconfigfile loaded successfully
Sonobuoy Version: v0.56.16
MinimumKubeVersion: 1.17.0
MaximumKubeVersion: 1.99.99
GitSHA: c7712478228e3b50a225783119fee1286b5104af
GoVersion: go1.19.10
Platform: linux/amd64
API Version:  v1.25.3
INFO:interface: invoke cncf conformance test
INFO[0000] create request issued                         name=sonobuoy namespace= resource=namespaces
INFO[0000] create request issued                         name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] create request issued                         name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] create request issued                         name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] create request issued                         name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] create request issued                         name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] create request issued                         name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] create request issued                         name=sonobuoy-aggregator namespace=sonobuoy resource=services
14:08:41          PLUGIN                        NODE    STATUS   RESULT   PROGRESS
14:08:41    systemd-logs   scs-cluster-control-plane   running                    
14:08:41             e2e                      global   running                    
14:08:41 
14:08:41 Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
...
...
14:09:41    systemd-logs   scs-cluster-control-plane   complete                    
...
14:10:21    systemd-logs   scs-cluster-control-plane   complete   passed                         
14:10:21             e2e                      global   complete   passed   Passed:960, Failed:  0
14:10:21 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
INFO:interface: 1094 passed, 5976 failed of which 5976 were skipped
INFO:interface:removing sonobuoy tests from cluster
INFO[0000] delete request issued                         kind=namespace namespace=sonobuoy
INFO[0000] delete request issued                         kind=clusterrolebindings
INFO[0000] delete request issued                         kind=clusterroles

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 2 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 1 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-09-19 14:10:26 +0200 CEST Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...
...
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
INFO:interface:removing sonobuoy tests from cluster
INFO[0000] already deleted                               kind=namespace namespace=sonobuoy
INFO[0000] delete request issued                         kind=clusterrolebindings
INFO[0000] delete request issued                         kind=clusterroles

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
INFO:root:Deleting cluster scs-cluster..
Deleting cluster "scs-cluster" ...

[mbuechse]

Please re-request review from me when you've reached the point.

[tonifinger]

Please re-request review from me when you've reached the point.

I have reached a point where you can test the first approach by using scs-test-runner.py to run the kaas tests on self provisioned k8s clusters.

You should be able to test this with the following command:

./scs-test-runner.py --config ./config-kaas-example.toml --debug run --preset="all-kaas" -o report.yaml --no-upload 

This PR is still in draft as I have to:

• Handle configurations to allow selecting cluster versions on plugin side
• Handle generic configuration hand over
• Add the function to delete clusters to the cleanup command
• Finalize static_plugin to handle excising k8s clusters
• Add integration Tests

[mbuechse]

Just a quick update on ongoing work:

• we decided that we create a 'subject' for each combination of CSP and k8s branch,
• we decided that each such subject must be equipped with the additional information necessary for the plugin in question to work (for instance, if the CSP uses Gardener, then we might need some sort of "shoot cluster manifest")
• if this additional information is added to this repository at all, it would either go into .zuul.d/secure.yaml or into somewhere under playbooks (and then referenced by config.toml)

[mbuechse]

Just a few remarks

[mbuechse]

This is starting to look really good now. A few minor things though.