feat: add vcluster e2e cluster provider

[crandles]

addition of vcluster provider; interested to see if the CI fails. vcluster needs a "host" cluster to launch into, so I've followed an existing example where a real cluster or kind cluster are utilized as necessary:

e2e-framework/examples/real_cluster/main_test.go

Line 39 in 9206e6f

if os.Getenv("REAL_CLUSTER") == "true" {

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR enables a new E2EClusterProvider for vcluster based clusters.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., Usage docs, etc.:

NA

[k8s-ci-robot]

Hi @crandles. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vladimirvivien]

@crandles this is awesome and thanks for the contribution.
At first glance it looks Okay, but I will ask @harshanarayana to take a look as well since he's the E2E provider expert 😉

[vladimirvivien]

This is a great start. I left some comments.

[vladimirvivien]

I think @harshanarayana has some util wrapper for running procs that should be considered here.

[vladimirvivien]

This looks good to me.
LGTM

[vladimirvivien]

/retest

[crandles]

Updated copyright, verify-boilerplate passes locally now.

[vladimirvivien]

/retest

[crandles]

golangci appears to be passing now for thew new code I've added, though errirng for some existing LOC:

third_party/helm/helm.go:240:9: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
		err = fmt.Errorf(missingHelm)
		      ^
third_party/flux/flux_setup.go:47:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
			return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
			            ^
third_party/flux/flux_setup.go:61:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
			return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
			            ^
pkg/env/env.go:229:11: printf: non-constant format string in call to (*testing.common).Skipf (govet)
		t.Skipf(message)
		        ^
pkg/env/env.go:513:22: printf: non-constant format string in call to (*testing.common).Skipf (govet)
					internalT.Skipf(message)

Assuming that won't block this PR.

[vladimirvivien]

@harshanarayana and @cpanato any idea why golangci failing on previously working code ?

[crandles]

errors were primarily for:

SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)

It's possible that I'm using a different toolchain version, but as the warning seemed fair, I've fixed it via cf9cbf2

make verify now passes locally.

[vladimirvivien]

@crandles is this ready to go now ?

[crandles]

Yes, I believe so

[crandles]

rebased due to go.mod conflict

[vladimirvivien]

LGTM

[vladimirvivien]

@harshanarayana @ShwethaKumbla @cpanato PTAL I think we need a second reviewer.

[harshanarayana]

I will check by eod tomorrow.

[crandles]

Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)

[harshanarayana]

Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)

@crandles

Helm and flux are installed slightly in a different way, https://github.com/kubernetes-sigs/e2e-framework/blob/777cbcd59fbbe572b71dde89c8f7b0b1a5ddc4a7/hack/install-helm.sh

test: install-helm install-flux ## Runs golang unit tests
	./hack/test-go.sh

But you are right about ko or kwok though

[vladimirvivien]

/retest

[crandles]

Is there a recommended way to simulate the prow test runner locally?

This looks relevant: https://docs.prow.k8s.io/docs/build-test-update/#running-a-prowjob-locally (and this job), I'll try to follow it.

---

Trying this out so far, not quite working yet:

# new makefile target
run-pr-tests:
	mkdir -p ./build ./build/out ./build/node_dir
	curl -s -o build/e2e-framework-presubmits.yaml https://raw.githubusercontent.com/kubernetes/test-infra/master/config/jobs/kubernetes-sigs/e2e-framework/e2e-framework-presubmits.yaml
	curl -s -o build/pj-on-kind.sh https://raw.githubusercontent.com/kubernetes-sigs/prow/refs/heads/main/pkg/pj-on-kind.sh
	curl -s -o build/prow-config.yaml https://raw.githubusercontent.com/kubernetes/test-infra/refs/heads/master/config/prow/config.yaml
	NODE_DIR=$(realpath build/node_dir) OUT_DIR=$(realpath build/out) CONFIG_PATH=$(realpath build/prow-config.yaml) JOB_CONFIG_PATH=$(realpath build/e2e-framework-presubmits.yaml) bash ./build/pj-on-kind.sh pull-e2e-framework-test

Output

latest: Pulling from k8s-prow/mkpj
Digest: sha256:6e71c2d9f675bce23136443cd8093fad59ed3344325d305963e89eadb7d3b6d5
Status: Image is up to date for gcr.io/k8s-prow/mkpj:latest
gcr.io/k8s-prow/mkpj:latest

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpj:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
time="2024-11-13T04:38:37Z" level=warning msg="empty -github-token-path, will use anonymous github client"
time="2024-11-13T04:38:37Z" level=info msg="Throttle(0, 0, [])" client=github
PR Number: 450
time="2024-11-13T04:39:20Z" level=info msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450)" client=github
time="2024-11-13T04:39:20Z" level=debug msg="Using GitHub REST API Version: 2022-11-28" client=github
time="2024-11-13T04:39:20Z" level=debug msg="Used token for request" cache-mode= client=github method=GET path=/repos/kubernetes-sigs/e2e-framework/pulls/450 throttled=true
time="2024-11-13T04:39:20Z" level=debug msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450) finished" client=github duration=419.387292ms
latest: Pulling from k8s-prow/mkpod
Digest: sha256:1a95d701f6cafae2c004088e47f15efc0b8e5b8d8fb6a23a2e2ccd94471eda46
Status: Image is up to date for gcr.io/k8s-prow/mkpod:latest
gcr.io/k8s-prow/mkpod:latest

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpod:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
time="2024-11-13T04:39:22Z" level=info msg="Generated build-id for job." build-id=1856557443136360448
time="2024-11-13T04:39:22Z" level=info msg="Pod-utils configured for local mode. Instead of uploading to GCS, files will be copied to an output dir on the node." out-dir=/Users/c/code/e2e-framework/build/out/pull-e2e-framework-test/1856557443136360448
For each of the following volumes specify one of:
 - 'empty' to use an emptyDir;
 - a path on the host to use hostPath;
 - '' (nothing) to use the existing volume source and assume it is available in the cluster
Volume "service": empty
Applying pod to the mkpod cluster. Configure kubectl for the mkpod cluster with:
>  export KUBECONFIG='/Users/c/.kube/kind-config-mkpod'
NAME                                   READY   STATUS     RESTARTS   AGE
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     Init:0/3   0          0s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     Init:0/3   0          4s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     Init:1/3   0          7s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     Init:2/3   0          10s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     Init:2/3   0          12s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   0/2     PodInitializing   0          13s
aed6b1e8-6ada-4b27-90ad-c7387846b7b0   2/2     Running           0          70s
---
% kubectl logs -f aed6b1e8-6ada-4b27-90ad-c7387846b7b0
Defaulted container "test" out of: test, sidecar, clonerefs (init), initupload (init), place-entrypoint (init)
Docker in Docker enabled, initializing...
================================================================================
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
Starting Docker: docker.
Waiting for docker to be ready, sleeping for 1 seconds.
================================================================================
Done setting up docker in docker.
ERROR: (gcloud.auth.activate-service-account) Unable to read file [/etc/service-account/service-account.json]: [Errno 2] No such file or directory: '/etc/service-account/service-account.json'
+ WRAPPED_COMMAND_PID=271
+ wait 271
+ make test
./hack/install-helm.sh
Downloading https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz
Verifying checksum... Done.
...
./hack/test-go.sh
go: downloading sigs.k8s.io/controller-runtime v0.19.1
go: downloading k8s.io/klog/v2 v2.130.1
go: downloading k8s.io/client-go v0.31.2
go: downloading k8s.io/apimachinery v0.31.2
...
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown.
...

[crandles]

found/resolved a couple of issues:

• I was mistaken on the vcluster installation path, it is vclusterctl not vcluster
  • Apparently I've been falling back to an alternatively installed vcluster in my CI
• there was a flaw in the version defaulting logic

[cpanato]

/ok-to-test

[cpanato]

thanks

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpanato, crandles, vladimirvivien

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cpanato,vladimirvivien]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[crandles]

Thanks for the reviews!

I will try to rebase next time, and I may follow up with an issue about how to run the pr tests locally. I need to spend more time looking at the pj on kind script.