Timeout waiting for process kube-apiserver to stop

[ysksuzuki]

What happened:

envtest.Environment.Stop() raises the error timeout waiting for process kube-apiserver to stop with kube-apiserver 1.21.2 that set up using setup-envtest use -p env.

  Unexpected error:
      <errors.aggregate | len:1, cap:1>: [
          <*errors.errorString | 0xc000328c00>{
              s: "timeout waiting for process kube-apiserver to stop",
          },
      ]
      timeout waiting for process kube-apiserver to stop
  occurred

How to reproduce it (as minimally and precisely as possible):

Run this envtest with kube-apiserver 1.21.2.

Environment:

• controller-runtime: 0.9.2
• kube-apiserver: 1.21.2

[stijndehaes]

using 1.20.2 for example works fine

[varshaprasad96]

+1, hit the same issue and using 1.20 binaries works fine.

[erikgb]

+1, I also tested with the 1.22 binaries, and still same issue.

It would be nice if someone with the skills could take a look ASAP. 😉

/priority important-soon

[joelanford]

I'm not able to reproduce this issue with the envtest 1.22.0 binaries and controller-runtime v0.9.6.

To try to reproduce, I bumped to the latest k8s, envtest, and c-r versions here

[hickeyma]

Tested using the cronjob-tutorial, results as follows:

• Kubernetes v1.20.2: No issue
• Kubernetes v1.21.4: Issue as stated
• Kubernetes v1.22.0: Issue as stated

So, looks like issue in Kubernetes v1.21, v1.22.

[hickeyma]

This is what I have found so far. From Kubernetes 1.21+, when it tries to cleanup the test environment, there is a clash if a custom controller is created during testing. It would seem that the controller is still running and kube-apiserver will not respond to shutdown during tear down.

I can get it to work by, changing controller goroutine:

go func() {
		err = k8sManager.Start(ctrl.SetupSignalHandler())
		Expect(err).ToNot(HaveOccurred())
}()

to:

go func() {
		defer GinkgoRecover()
		err = k8sManager.Start(ctrl.SetupSignalHandler())
		Expect(err).ToNot(HaveOccurred(), "failed to run manager")
		gexec.KillAndWait(4 * time.Second)

		// Teardown the test environment once controller is fnished.
		// Otherwise from Kubernetes 1.21+, teardon timeouts waiting on
		// kube-apiserver to return
		err := testEnv.Stop()
		Expect(err).ToNot(HaveOccurred())
}()

and disabling the teardown in AfterSuite.

[tenstad]

Is it a mistake that the ctx with cancel in suite_test.go#L65 is replaced by context.Background() in suite_test.go#L97?

[erikgb]

Thx @tenstad! I think this issue can ble closed, as not a problem (here), ref. cybozu-go/coil#189.

I will prepare a PR to fix the skaffolding (and doc) in kubebuilder, ref. the workaround implemented in kubernetes-sigs/kubebuilder#2302. CC: @hickeyma