fix: make kind-with-registry.sh work with podman

[wbrefvem]

Addresses #3729

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wbrefvem
Once this PR has been reviewed and has the lgtm label, please assign bentheelder for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @wbrefvem!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @wbrefvem. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

isn't the right fix to make kind get nodes work with podman?

[aojea]

podman works with kind get nodes, it logs the experimental disclaimer ... podman is not stable enough and still lacks some important capabilities #1778

[stmcginnis]

There's some small discussion on the linked issue that highlights why the current kind get nodes output is a challenge with podman.

[aojea]

@wbrefvem I didn't realize, but the script does not rely on kubectl commands, since you can not guarantee you are targeting the right cluster

Since all kind nodes have a -control-plane or -worker suffix we may use that inside the loop to skip those, WDYT @BenTheElder

[aojea]

oh, my bad again, it does rely on kubectl at the end of the script ... nevermind, let me reassing this to @BenTheElder , he knows better this script

/assign @BenTheElder

[BenTheElder]

It should be stderr, which should not be captured:

kind/pkg/internal/runtime/runtime.go

Line 16 in 0483056

logger.Warn("using podman due to KIND_EXPERIMENTAL_PROVIDER")

kind/pkg/cmd/logger.go

Lines 31 to 36 in 0483056

	func NewLogger() log.Logger {
	var writer io.Writer = os.Stderr
	if env.IsSmartTerminal(writer) {
	writer = cli.NewSpinner(writer)
	}
	return cli.NewLogger(writer, 0)

[BenTheElder]

kind/pkg/cluster/internal/providers/podman/provider.go

Line 45 in 0483056

logger.Warn("enabling experimental podman provider")

[BenTheElder]

$ (out="$(KIND_EXPERIMENTAL_PROVIDER=podman kind get nodes || true)"; echo "out: $out")
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to list nodes: command "podman ps -a --filter label=io.x-k8s.kind.cluster=kind --format '{{.Names}}'" failed with error: exec: "podman": executable file not found in $PATH
Command Output: 
out: 

Are we sure the script is broken?

[wbrefvem]

Are we sure the script is broken?

For podman running on macOS, yes, but I may have misunderstood the reason. Here's what I'm seeing when the script starts to docker exec the nodes:

using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Error: can only create exec sessions on running containers: container state improper

@BenTheElder you're right that the two lines about podman are not going to stdout. It must be a race condition, and since my solution involves an API call, it's slow enough to lose the race.

[BenTheElder]

Interesting, we really should be safe to exec once kind create cluster succeeds ... uhhh

Did the container crash or something?

Let's follow up more on the issue.

[BenTheElder]

This seems to be accidentally mitigating some deeper issue, closing the PR for now but watching the issue for more details so we can determine the root cause and fix