improve "kubeadm reset" kube-proxy cleanup

[danwinship]

@pacoxu pointed out the kubeadm reset kube-proxy cleanup instructions in kubernetes/kubernetes#128886. I started writing a better version:

diff --git a/cmd/kubeadm/app/cmd/reset.go b/cmd/kubeadm/app/cmd/reset.go
index 5fdc7ec193f..dbfede5c695 100644
--- a/cmd/kubeadm/app/cmd/reset.go
+++ b/cmd/kubeadm/app/cmd/reset.go
@@ -47,13 +47,23 @@ import (
 )
 
 var (
-	iptablesCleanupInstructions = dedent.Dedent(`
-		The reset process does not reset or clean up iptables rules or IPVS tables.
-		If you wish to reset iptables, you must do so manually by using the "iptables" command.
+	kubeProxyCleanupInstructions = dedent.Dedent(`
+		The reset process does not reset or clean up rules created by kube-proxy.
 
-		If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
-		to reset your system's IPVS tables.
+		If you wish to reset iptables rules created by kube-proxy in "iptables"
+		or "ipvs" mode, you must do so manually by using the "iptables" command.
 
+		If you were running kube-proxy in "ipvs" mode, run
+		    ipvsadm --clear
+		(or similar) to reset your system's IPVS tables.
+
+		If you were running kube-proxy in "nftables" mode, run
+		    nft delete table ip kube-proxy
+		    nft delete table ip6 kube-proxy
+		to remove kube-proxy's nftables rules.
+	`)
+
+	kubeconfigCleanupInstructions = dedent.Dedent(`
 		The reset process does not clean your kubeconfig files and you must remove them manually.
 		Please, check the contents of the $HOME/.kube/config file.
 	`)
@@ -236,8 +246,10 @@ func newCmdReset(in io.Reader, out io.Writer, resetOptions *resetOptions) *cobra
 
 			// output help text instructing user how to remove cni folders
 			fmt.Print(cniCleanupInstructions)
-			// Output help text instructing user how to remove iptables rules
-			fmt.Print(iptablesCleanupInstructions)
+			// Output help text instructing user how to remove kube-proxy rules
+			fmt.Print(kubeProxyCleanupInstructions)
+			// Output help text instructing user how to remove kubeconfigs
+			fmt.Print(kubeconfigCleanupInstructions)
 			return nil
 		},
 	}

But it seems weird and awkward that we explain how to clean up nftables and the ipvs half of ipvs mode, but we don't explain how to clean up iptables (including the iptables half of ipvs mode). Plus, kubeadm is no longer requiring the user to have the iptables binaries installed any more anyway...

kube-proxy already has a --cleanup flag that will clean up all iptables, ipvs, and nftables rules. Maybe it would be nicer to tell the user to run something like:

docker run -it --network=host ${KUBE_PROXY_IMAGE} kube-proxy --cleanup

(not tested). Or maybe something with crictl (except obviously the user isn't going to have that installed). Or maybe kubeadm reset could (optionally?) run the pod itself via CRI? Or by running kubelet in standalone mode just long enough to run one pod? I'm not sure what would be easiest for kubeadm...

Anyway, feel free to just take the patch above if you want the simple fix.

[pacoxu]

kubernetes/kubernetes#126596 we don't depends on crictl as well after v1.31.

• For suggestions, crictl, docker or nerdctl are similar. I prefer docker as it can work. crictl will not work for run -it.

[neolit123]

@danwinship
i don't think we should print any step-by-step docs in the reset output. we could show a link to a page hosted at k/website that instructs users how to do the cleanup with the --cleanup flag.

i think kube-proxy needs to:

• document its dependencies
• document how to use the --cleanup flag or how to do manual cleanup for various modes

kubernetes/kubernetes#126596 we don't depends on crictl as well after v1.31.

• For suggestions, crictl, docker or nerdctl are similar. I prefer docker as it can work. crictl will not work for run -it.

yes, kubeadm now interacts directly with the CRI. 'reset' can create a container and call kube-proxy with the --cleanup flag, but some users of kubeadm don't deploy kube-proxy, so that would mean also checking if the kube-proxy config map exists. i think this is doable, but i don't have bandwidth to play with that. also, i think it may step over the line of the 'best-effort' principle of 'reset' i.e. we consider 'reset' to be a best effort cleanup of the host - some things are skipped, some things may fail.

cc @SataQiu @carlory
for additional comments.

[carlory]

i don't think we should print any step-by-step docs in the reset output. we could show a link to a page hosted at k/website that instructs users how to do the cleanup with the --cleanup flag.

+1. A document link is preferred.

[SataQiu]

also +1 to link to a website page
we just need to keep the web page up to date.

[neolit123]

putting 1.33 milestone on this, but there is nothing actionable for kubeadm ATM.
kube-proxy / sig network would need to write the page we proposed above.

[danwinship]

i don't think we should print any step-by-step docs in the reset output. we could show a link to a page hosted at k/website that instructs users how to do the cleanup with the --cleanup flag.

We don't currently document the flag anywhere other than the autogenerated man page, but there's barely anything to document: you just run kube-proxy --cleanup. That's it. You don't need to pass a config file or any other options, because it just ignores all the configuration and does a best-effort cleanup of all backends.

some users of kubeadm don't deploy kube-proxy, so that would mean also checking if the kube-proxy config map exists

though fwiw kube-proxy --cleanup should be a no-op if you didn't run kube-proxy; it would just be trying to delete a bunch of iptables chains and ipvs servers and nftables tables that don't exist.

document how to use the --cleanup flag or how to do manual cleanup for various modes

kube-proxy --cleanup is the supported way to clean up. Any more-specific documentation is likely to get out of date over time.