✨ Enable using setup-envtest without a separate CLI

[tomasaschan]

This implements the proposal in #2790.

The implementation is incomplete so far; opening this mostly to show my progress and solicit feedback, as well as to avoid too many conflicts with initiatives like this one.

To do:

• Copy existing implementation of supporting packages over into the main module
• Implement use in the main module
• Implement list in the main module
• Implement cleanup in the main module
• Implement sideload in the main module
• Port tests for the workflows (at least the ones that make sense)
• Clean up the setup-envtest module, removing duplicated code etc
• Remove dependency on afero
• Rebase onto whatever latest changes have happened in e.g. go.mod since I branched off
• (Optional, possibly in a later PR) Re-implement the arg parsing and help parts of the main module with something a little less home-grown, probably something like https://github.com/spf13/cobra/

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tomasaschan
Once this PR has been reviewed and has the lgtm label, please assign fillzpp for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tomasaschan]

I will rebase this properly once #2811 merges, since I expect conflicts with that anyway.

[tomasaschan]

@sbueringer I think this is ready for an initial review now!

I still want to get rid of the dependency on afero before we merge this, so I'm keeping the PR in a draft state, but those changes should be basically independent of the major ones here. Let me know if you have any questions or would like me to somehow document the structure of the new set of packages to make reviewing easier.

[tomasaschan]

I've spent some time now trying to rebuild this without a dependency on github.com/spf13/afero, and while I was initially encouraged by the existence of testing/fstest in the standard libarry, it turns out that while the afero implements a complete file system abstraction (and implementations for the two pieces we need; an OS wrapper and an in-memory one for testing), fstest only implements file actions for a read-only file system, expecting the tests to set up the appropriate input data before executing the SUT.

While it's certainly possible to build a read/write file system abstraction (on top of fstest or standalone) that supports the operations we need for setup-envtest, I feel it's probably not worth it in maintenance cost, compared to just relying on the third-party library. afero does not change rapidly and so we're unlikely to cause dependency hell issues by taking a dependency on it.

@sbueringer What do you think?

If you agree with this assessment, I can rebase again to get the latest version bumps of everything, and then this should be ready to go!

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sbueringer]

@tomasaschan Just a heads up. We'll merge a PR directly after 0.19 which drops the support for GCS. Basically because we wanted to do it anyway, but also to simplify maintenance & refactorings like this PR

[tomasaschan]

Thanks for the heads up!

Have you had time to review these changes, and in particular my last comment about keeping the afero dependency?

[sbueringer]

Didn't get around to it yet. Just recently found time to catch up on controller-runtime in general and this is a big PR. But I'll try to look at it soon. Definitely feel free to wait with further rebases until then

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sbueringer]

/remove-lifecycle stale

[tomasaschan]

/lifecycle frozen

@sbueringer and potentially other maintainers: given the initial hesitation to introduce a dependency on github.com/spf13/afero and my analysis of the available options for properly testing this (see #2810 (comment)), I'd like someone to let me know what will be required for this to be merged before I undertake the non-trivial effort to rebase this.

[k8s-ci-robot]

@tomasaschan: The lifecycle/frozen label cannot be applied to Pull Requests.

In response to this:

/lifecycle frozen

@sbueringer and potentially other maintainers: given the initial hesitation to introduce a dependency on github.com/spf13/afero and my analysis of the available options for properly testing this (see #2810 (comment)), I'd like someone to let me know what will be required for this to be merged before I undertake the non-trivial effort to rebase this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tomasaschan]

@sbueringer Any chance I can get a more explicit decision on afero than a 👍 on my question above? 😅

[sbueringer]

@tomasaschan Sorry should have brought this up way earlier but somehow I didn't think about that.

What is the functionality we are actually looking for from an envtest point of view? Download & extract the binaries if they don't already exist locally?

Just seems that if that's mostly what we are looking for, setup-envtest has an insane level of complexity.

Also it's probably a lot easier to add a few functions that cover what we need to the envtest package (they can reuse e.g. the http_client underneath) than porting all the commands and workflows over.

And sorry I think I lost the overview a bit, the idea of the current PR was to basically move the entire setup-envtest implementation into the main module with some refactorings so that everything can be also called in test code and not only via the CLI, correct?

Regarding afero. It has dependencies on e.g. go.opentelemetry.io/otel/*. As soon as we don't find compile compatible combinations of k8s.io/* and afero because of transitive otel dependencies we would have to drop the code using afero.

[tomasaschan]

@sbueringer My motivation for this PR is outlined in the issue linked from the PR body. In short, I'd like to avoid the need for wrapping the call to go test ./... with calls to setup-envtest to ensure that binaries are both available to (the ie downloaded) and found by (typically through setting an env var) envtest. Concretely, I want to enable things like kubebuilder projects to not have to use make test for running tests.

Wrt afero, the main reason to use it in setup-envtest is to make the file-system dependent parts testable; afero provides an abstraction that has both file system backed and in-memory backed implementations, meaning it's relatively easy to write tests that verify the file system dependent operations (anything from downloading the right version to choosing the right one based on what's already locally available). I did survey for similar abstractions in the stdlib, but found none - the one that exists is basically read-only.

What if the change instead involved pulling envtest out of controller-runtime (rather than pulling setup-envtest into it)? That way, dependencies would not be such an important consideration (depending on envtest is much more optional that depending on all of controller-runtime), and users of envtest could still configure their test harness to run without ad-hoc wrapping scripts. Would that be a more acceptable approach?

[sbueringer]

What if the change instead involved pulling envtest out of controller-runtime (rather than pulling setup-envtest into it)?

This would only help if controller-runtime itself stops using envtest for its own testing. Otherwise we end up with the same transitive dependencies.

If I look at this from an envtest perspective, we basically have to invoke HTTPClient.GetVersion and then add a bit of code around it to ensure the file is written to the right location (+ skip if it's already there), right?

I think testing for this can be done without abstracting away the filesystem.

[tomasaschan]

If I look at this from an envtest perspective, we basically have to invoke HTTPClient.GetVersion and then add a bit of code around it to ensure the file is written to the right location (+ skip if it's already there), right?

Plus all of the parsing of desired version, and resolving that (which involves looking at what's available locally, potentially at a custom location, and online, potentially with a custom registry). Additionally, there's still a few places where error handling is basically panic, which I suppose would have to be refactored for it to really be usable in a test context. (Although this is much rarer now than it used to be, nice! 👍🏻) To "add a bit of code around it" would essentially re-implement parts of the Store if we don't just use that, so it feels much more reasonable to make sure that part of setup-envtest can also be made available from a package.

I think testing for this can be done without abstracting away the filesystem.

The main benefit of in-memory file systems for tests over real ones, IMO, is that the in-memory ones share no state, so there is no coupling between the tests. If we write the tests to interact with a real file system, we'll have all kinds of issues around shared state, test ordering, etc. Do you have a proposed way to avoid that?

[sbueringer]

The main benefit of in-memory file systems for tests over real ones, IMO, is that the in-memory ones share no state, so there is no coupling between the tests. If we write the tests to interact with a real file system, we'll have all kinds of issues around shared state, test ordering, etc. Do you have a proposed way to avoid that?

Make paths configurable and use different tmp folder(s) for different tests