Investigate kind replacement for e2e

[a7i]

Is your feature request related to a problem? Please describe.

Descheduler release are typically blocked by waiting for a kind node image.

Describe the solution you'd like

1. Either build our own kind node image
2. Pursue alternatives such as k3d or minikube, which keep up with latest versions

Describe alternatives you've considered

Stay behind on releases

Additional context

kubernetes-sigs/kind#3589

[BenTheElder]

As an upstream kubernetes project you should really consider testing against upcoming Kubernetes before it releases, by doing kind build node image within the e2e pipeline and then kind create cluster --image=kindest/node:latest (from the locally built image, more in the docs).

EDIT: 1.30 is out anyhow.

[damemi]

Thanks @BenTheElder, that's a great suggestion and I think it's exactly what we've been looking for. It has always felt a bit weird that we were dependent on waiting until after the kind release to test and publish our next release. I didn't know about that option

[a7i]

cc: @pravarag since you wanted to look into this.

I think we could do something like:

docker pull kindest/node:v1.30.0 || build_kind_image_from_source

[damemi]

@a7i I'd say our master branch could always be building from source, with our tag branches using the released version

The downside to that is we run the risk of getting master blocked on bugs from kind though

[a7i]

Wouldn't that still get us blocked on kind image being released? Unless that's intentional?

[damemi]

It would block new PRs to the tagged branch until that image was available, so maybe the release branches could use the conditional switch like you're suggesting

[BenTheElder]

For K/K we're fetching kind from HEAD to stay compatible with any kubernetes breaking changes, but we're also running equivalent CI jobs, it's possible we'd have a breaking change that you'd have found in the release notes.

That said, we're avoiding those as much as possible, and when we're planning one they've generally been pre-announced (like the containerd 2.0 style registry config) in previous release notes similar to Kubernetes style deprecations. This isn't always possible when we have to react to e.g. the runc misc cgroup changes but generally speaking upgrading to those changes should be desired in a typical CI environment.

We're discussing continuous image builds in the future, right now we build releases with https://github.com/kubernetes-sigs/kind/blob/0a7403e49c529d22cacd3a3f3606b9d8a5c16ae7/hack/release/build/push-node.sh which amongst other things is making the images smaller by compiling out dockershim and cloud providers ... which won't be necessary vs standard builds in 1.31+

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pravarag]

/remove-lifecycle stale

[pravarag]

@a7i apologies on letting this issue slip for this long. I've started investigating on it again and will be sharing my further inputs/doubts on it soon. Thanks!

[a7i]

Thanks @pravarag , no worries I totally understand how open-source goes and we certainly appreciate your contribution.

the images are published quickly now too so it's not as urgent: https://hub.docker.com/r/kindest/node/tags

[BenTheElder]

FYI you can also cheaply build images from kubernetes release binaries (recommended usage is only for 1.31+ but it works for older releases too, just larger images)

multi-arch support isn't there yet, but you could combine them with docker manifest if you require that.

For just quickly building an image at a custom version to use in github actions, see the release notes at https://github.com/kubernetes-sigs/kind/releases/tag/v0.24.0 / https://kind.sigs.k8s.io/docs/user/quick-start/#building-images

This is much faster and doesn't require compiling, only downloading a build.