Add conformance tests for scs-0210-v2 version policy

[martinmo]

This adds the missing conformance tests for scs-0210-v2. It has still some rough edges and some of the code doesn't look pretty, but I think it is good enough to be used.

One thing that needs to be maintained from now on, until we find a better solution, is the new k8s-eol-data.yml file which contains the EOL data of the K8s release branches.

To test it, provide a kubeconfig that has the contexts stable, oldstable and oldoldstable. You can use minikube to create clusters of different versions, e.g., minikube create -p stable --kubernetes-version=... and provide -k ~/.kube/config as a script argument.

[mbuechse]

I will have a look. At first glance, it looks really good!

[mbuechse]

Again, it looks really good. Does it test though that the three contexts actually use different K8s versions? I missed that.

[martinmo]

Again, it looks really good. Does it test though that the three contexts actually use different K8s versions? I missed that.

Yes, it does so by collecting the used "branches" in a set and checking them afterwards. (Release) Branch means tuple(major, minor) in K8s context. Remark: I used the word "branches" here for two reasons: the K8s release docs use it and there is no confusion with the minor version.

[martinmo]

I've finished the last few missing bits. The PR description is updated and the draft status removed. Please review. If someone other than me merges this, feel free to clean up the squash merge message.

[cah-hbaum]

LGTM!
Tested at least with the "stable" part of the cluster and that worked well.

[mbuechse]

As I said, I like it very much. A few remarks here and there, which could be converted into an issue for later. Most importantly: try to find as many problems with the clusters as possible before returning to the caller.

[mbuechse]

@martinmo And one question that just occurred to me: are these hard-coded context names practicable? Or could we allow for a more sophisticated scheme? If we knew that three versions are officially supported (which is not always the case), then the command-line argument could look like the following: -k kubeconfig.yaml:stable -k kubeconfig.yaml:oldstable -k kubeconfig.yaml:oldoldstable. What do you think about that?

edit Addendum: maybe this scheme is not compatible with the variables mechanism in the certificate-scope yaml. But something like that.

[mbuechse]

Please fix this bug that was taken over from the old program: #492 (comment)

[martinmo]

@martinmo And one question that just occurred to me: are these hard-coded context names practicable? Or could we allow for a more sophisticated scheme? If we knew that three versions are officially supported (which is not always the case), then the command-line argument could look like the following: -k kubeconfig.yaml:stable -k kubeconfig.yaml:oldstable -k kubeconfig.yaml:oldoldstable. What do you think about that?

I like your idea. We then could simplify some of the code, especially I could remove that "skipped because next context references a cluster that is already EOL" step.

Do you intend to change the script call in Tests/scs-compatible-kaas.yaml on a regular basis, e.g., when there is a switch from 4 to 3 simultanous supported K8s upstream branches (or vice versa)?

[mbuechse]

Do you intend to change the script call in Tests/scs-compatible-kaas.yaml on a regular basis, e.g., when there is a switch from 4 to 3 simultanous supported K8s upstream branches (or vice versa)?

@martinmo I think these specifics are best captured using a variable (just like the kubeconfig itself). We have one Zuul job per subject, and I would set the variable there (as a Zuul/Ansible variable that can then be passed on to the main check script, which in turn it passes it on to the individual script).

We shouldn't forget though that in the future we are supposed to create the clusters ourselves. So this could already be fashioned with plugins in mind. So the actual syntax could be --cluster kubeconfig:kubeconfig.yaml:v1.29, where kubeconfig is fixed, and later we could add new methods such as --cluster gardener:XYZ or --cluster CAPI:xyz. Not quite sure.

[mbuechse]

@martinmo One more thing: Should we just ignore any cluster whose branch is no longer supported? Maybe this could facilitate the transition periods. Otherwise people have to change the invocation of the script on the day of EOL.

[martinmo]

@martinmo One more thing: Should we just ignore any cluster whose branch is no longer supported? Maybe this could facilitate the transition periods. Otherwise people have to change the invocation of the script on the day of EOL.

This is already done. One important caveat though is that the contexts need to be checked from newest to oldest for this detection to work properly.

[martinmo]

I incorporated your suggestions (thx @mbuechse and @cah-hbaum) and reworked the error handling, which now works in a similar fashion to the entropy check. But still, critical errors need to return early.

Btw, TIL that it's also possible to be SCS-compliant with only two supported K8s versions, at least as of today: v1.27 and v1.28. This is because v1.26 is EOL and v1.29 is still in the 4 month cadence period.

[martinmo]

One thing that bothers me: this file needs to be maintained from now on. I couldn't find a machine readable version upstream, just the Patch releases document.

[mbuechse]

We could assume that this rhythm of Oct 28, Feb 28, June 28 goes on, and then we only need to change anything if this rhythm breaks. Then of course we also need to know when these versions come out initially.

[martinmo]

Okay. With this scheme, another assumption that can break is that only the minor branch increments. I would like to take that idea into a separate issue.

For this PR, what do you think of issuing a warning if the file seems to be out of date at the time of script execution. This is the case if the file lists less than 3 non-EOL releases. And bail out if there are less than 2. With that approach we would at least see it if we monitor the logs in the CI.

[martinmo]

issuing a warning if the file seems to be out of date at the time of script execution. This is the case if the file lists less than 3 non-EOL releases. And bail out if there are less than 2.

I've implemented this in the last commit. If you do not agree with that change, I can easily revert it, no problem :-)

[martinmo]

@martinmo I think these specifics are best captured using a variable (just like the kubeconfig itself). We have one Zuul job per subject, and I would set the variable there (as a Zuul/Ansible variable that can then be passed on to the main check script, which in turn it passes it on to the individual script).

That makes sense.

We shouldn't forget though that in the future we are supposed to create the clusters ourselves. So this could already be fashioned with plugins in mind. So the actual syntax could be --cluster kubeconfig:kubeconfig.yaml:v1.29, where kubeconfig is fixed, and later we could add new methods such as --cluster gardener:XYZ or --cluster CAPI:xyz. Not quite sure.

Did you mean --cluster kubeconfig:kubeconfig.yaml:stable (instead of :v1.29 at the end)? Also, the examples remind me of connection strings :-) We could use URL syntax kubeconfig://./kubeconfig?context=stable or something and parse that with urlparse from urllib.parse.

[mbuechse]

@martinmo Yes, we could also use an URL syntax. I deliberately put v1.29 there instead of stable to indicate that providers could name their stuff whatever they like, and the meaning of stable would seem to me to drift over time, which I personally wouldn't like too much, but evyerone's mileage may vary.