Kubernetes version e2e test doesn't set versions correctly

[nojnhuh]

Bug description: The e2e tests for different Kubernetes versions defined in tests/e2e/e2e_k8s_version_test.go doesn't correctly set Kubernetes versions such that the tests actually run on kind clusters using those versions.

CI logs report a version matching kind's default used for both tests:

• https://github.com/openservicemesh/osm/runs/5855121651?check_suite_focus=true#step:11:506
• https://github.com/openservicemesh/osm/runs/5855121651?check_suite_focus=true#step:11:612

Currently, even if the -kindClusterVersion flag is passed, the framework always chooses kind's default because, based on my understanding of Ginkgo, the lines setting and resetting the cluster version are run back-to-back before the code inside the It block between them. Moving those inside the It block doesn't fix the issue because then Td.ClusterVersion is being set after the cluster is already provisioned by the framework.

Affected area (please mark with X where applicable):

• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [ ]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [ ]
• Logging [ ]
• Debugging [ ]
• Tests [X]
• Demo [ ]
• CI System [ ]

Expected behavior: The specified Kubernetes versions are used for those tests.

Steps to reproduce the bug (as precisely as possible):
Roughly matching the flags used in CI:

make test-e2e CTR_REGISTRY=localhost:5000 CTR_TAG=latest E2E_FLAGS="-installType=KindCluster -ginkgo.focus='v1.22.8'"

Logs then show:

> k8s server version: v1.23.4

And docker ps shows kindest/node:v1.23.4 running.

Bug report archive:

Environment:

• OSM version (use osm version):

CLI Version: version.Info{Version:"dev", GitCommit:"a918abff99a2106f913302db2fc8705651d2a72d", BuildDate:""}

• Kubernetes version (use kubectl version):
• Size of cluster (number of worker nodes in the cluster):
• Others:

[shashankram]

Good catch @nojnhuh. Would it make sense to override the default in the test, reset the version immediately after cluster install, and use the default version otherwise?

[nojnhuh]

I think the root of the problem is that these tests require their own specific cluster setups but CI is configured to run each bucket of tests entirely in one cluster. The framework has a -cleanupKindClusterBetweenTests that we can maybe use to get this working, but otherwise the framework wasn't really designed to handle tests like this that are tightly coupled to specific cluster configurations.

[shashankram]

I think the root of the problem is that these tests require their own specific cluster setups but CI is configured to run each bucket of tests entirely in one cluster. The framework has a -cleanupKindClusterBetweenTests that we can maybe use to get this working, but otherwise the framework wasn't really designed to handle tests like this that are tightly coupled to specific cluster configurations.

That explains the behavior. I forgot that in the CI we reuse the cluster between tests. Not sure if it's worth bucketing per version just for 3 tests. An alternative to getting coverage for at least 2 versions is to ensure the demo (maestro) and e2e run against different versions.