[Feature Request] CNCF conformance tests

[tonifinger]

This issue was created to provide a discussion ground for possible future standards. It is derived from SovereignCloudStack/issues#181 and one of the points not assigned any issue yet.

As a cloud service provider, I want my KaaS to comply with the Certified Kubernetes Software Conformance
Here we discuss how the "Certified Kubernetes Software Conformance" can be utilized.

Needs:

• [Feature Request] enable compliance tests to use plugins for cluster provisioning #710

Definition of Done:

• Perform CNCF conformance tests using sonobuoy Feat/sonobuoy integration #819
• Collecting the results and converting them into an output that can be interpreted by the SCS

[tonifinger]

Reasearch:

Following the short guide, one can achieve CNCF certification by passing certain tests and submitting the test results to the CNCF Conformance Repo on GitHub (https://github.com/cncf/k8s-conformance, Requierments: Certified_Kubernetes_Terms.md)

The tests themselves can be carried out according to these instructions: https://github.com/cncf/k8s-conformance/blob/master/instructions.md .

Following these instructions, the tests are executed by sonobuoy, which furthermore executes the kubernetes e2e tests.

The SCS itself already uses these tests and tools within the k8s-cluster-api-provider
The implementation of the tests is therefore already done when using the k8s-cluster-api-provider.
In addition, I would suggest making it standard for the SCS KaaS to successfully pass the CNCF conformance tests.

List of certified products:

The CNCF provides a list of products that have earned a "certified kubernetes status"
( see: https://www.cncf.io/training/certification/software-conformance/#logos)

This list is divided into the following sections:

• Platform
• Certified Kubernetes - Distribution
• Certified Kubernetes - Hosted
• Certified Kubernetes - Installer
• PaaS/Container Service

Perhaps the SCS could achieve to be listed in any of the products categories?
Otherwise, we could write a decision dataset for users on what they need to do to get their SCS- deployed KaaS infrastructure certified by the CNCF.

---

SCS Implementation(draft):

Useful links:

• cluster stack continuous-integration.md
• example of zuul job result
• k8s-cluster-api-provider .zuul.yaml
• Zuul config repository

[mbuechse]

@tonifinger See v2 here: https://docs.scs.community/standards/scs-compatible-kaas
The question is: how do we transfer the results of the e2e pipeline into the compliance check tool?

[tonifinger]

@tonifinger See v2 here: https://docs.scs.community/standards/scs-compatible-kaas The question is: how do we transfer the results of the e2e pipeline into the compliance check tool?

To transfer the results of job/k8s-cluster-api-provider-e2e-conformance to the Python script scs-compliance-check.py we need to accomplish the following actions:

As a first step, the result of the conformance check is generated by the following “Zuul Job”: k8s-cluster-api-provider-e2e-conformance. ( The job executes the following playbook: playbooks/tasks/sonobouy.yaml )

Hint:
The mode used in the Zuul job configuration is conformance (see: .zuul.yaml#L32)
However in order to submit a result, according to "How to submit conformance results" we must use the mode certified-conformance

The generated data holding the results must be transferred to the "zuul job" running the "scs-compliance-check.py" script: k8s-cluster-api-provider-scs-compliance-1.27. ( The job executes the following playbook: playbooks/tasks/scs_compliance.yaml )
To accomplish this, we could use “zuul return-values”.

Finally, to check the results, the test script must be extended to analyze the test results generated by sonobuoy.

Hint:
This goes hand in hand with the use of sonobouy as a test tool for the "kaas scs compliance tests": see kaas-sonobuoy-go-example-e2e-framework .
We could use the same mechanism twice:
First to analyze the "e2e conformance" test results
Second to analyze the "kaas scs compliance tests" results

---

Question: Do we just want to test whether we would archive a certificate of compliance or do we also want to provide a mechanism to handle the process of archiving a certificate?

[mbuechse]

@tonifinger I don't understand the question at the end, but I will try to explain what is desired here to the best of my abilities.

For every certificate scope (such as "SCS-compatible KaaS"), we specify which tests have to be passed in order for the certificate to be awarded. This process is documented in scs-0003-v1.

A straightforward approach would be to write a Python script that "just" runs the CNCF tests, waits until they are finished, and then outputs the result (either "PASS" or "FAIL") -- this script could be included into our certificate scope just like every other test script that we are using.

The downside of this straightforward approach is the following: the CNCF tests are really quite time-consuming. In fact, taking at least 2 hrs, they dwarf all the other tests we do for our certificate scope (and this might be the case even for tests that we might add in the future).

I see three options of dealing with this downside:

1. Just live with it.
2. Kind of like 1, but declare the lifetime of the test result for the CNCF test to be a whole week, and make sure that this test is only run weekly. The SCS infrastructure does provide an option for that.
3. Have the CNCF tests run independently of the SCS tests. Then add a "dummy" test to the SCS suite that merely collects the test result of the CNFC tests somehow.

I think you are referring to this third option. It does have the distinct advantage that it would probably be easiest to implement with the SCS ClusterStacks implementation. AFAICT the e2e playbook is quite involved with creating a cluster etc., and this is something I wouldn't want to replicate (as would be necessary for Options 1 and 2).

We might be able to add a task to https://github.com/SovereignCloudStack/k8s-cluster-api-provider/blob/main/playbooks/tasks/sonobouy.yaml that takes the parsed results, writes them to a dummy file sono-result, and then executes scs-compliance-check.py with a certain parameter so that it would only run the above-mentioned dummy test (-t cncf-k8s-conformance, see here), which in turn would read the dummy file sono-result and report the result in the correct way. Then we could upload the results yaml (generated by scs-complience-check.py) the same way we do it in this playbook -- lo and behold, the result would be registered with the SCS compliance monitor, independently of all the other compliance tests.

There is such a big BUT there. The whole e2e business only works with the SCS reference implementation. So for any partner who uses something else, say, Gardener, we need some other way of running the CNFC tests, recording the results, and then upload them to the compliance monitor... I think we urgently need to get our hands on more varied environments, see #649 -- I'm afraid this has been stagnant for quite a while. Ideally, we could have an abstraction that allowed us for creating a cluster and running sonobouy on it, and then create concrete instances, one for ClusterStacks, one for Gardener, and so on... This would be an issue of its own, and maybe we should wait for that? This is definitely something that ought to be discussed in Team Container. Specifically: What would be a good way to create a K8s cluster that works on any partner cloud, regardless of the implementation they use?

[tonifinger]

DRAFT:
@mbuechse Thank you for your detailed answer!

First I would like to explain my above question in more detail.
I was referring to the final process of obtaining/announcing a certificate.
AFAIU, this can only be archived if you submit your results to the corresponding CNCF repository via a PR.

see: https://www.cncf.io/training/certification/software-conformance/#how
For example gardener uses a bot that automatically creates PR for the CNFC k8s-conformance repository
see: cncf/k8s-conformance#3315

So it should be possible to automatically create a PR to announce a certification.
My question is whether we also want to include this process in our pipeline or whether we want to leave the final process of submitting the results to the CNCF repository to the end user of the SCS?

---

Secondly, about the different approaches. Thanks for the evaluation of the different ways to approach this problem!
I'm currently figuring out how to reconcile this with #649.
I will write a more detailed response once I have a clear overview of this topic.