Add options like WaitUntilDeleted to Cleanup()

[pmalek]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

As it's done currently, Cleaner.Cleanup() waits for all the namespaces that it deletes, to actually get deleted.

This might sometimes be problematic e.g. in KIC's e2e tests we have a 3 minute context timeout for the cleanup and for some reason on GKE 1.25 it can't complete in time:

CI workflow: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/4180632755
CI job failure: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/4180632755/jobs/7241809673

Wed, 15 Feb 2023 04:54:12 GMT
    all_in_one_test.go:69: waiting for route from Ingress to be operational at http://34.71.10.27/httpbin
Wed, 15 Feb 2023 04:54:22 GMT
    helpers_test.go:85: INFO: cluster e2e-940b6b4a-f110-4038-9f6f-938b1f3da2f7 is being deleted
Wed, 15 Feb 2023 04:57:22 GMT
    helpers_test.go:85: 
Wed, 15 Feb 2023 04:57:22 GMT
        	Error Trace:	/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/teardown.go:29
Wed, 15 Feb 2023 04:57:22 GMT
        	            				/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/helpers_test.go:85
Wed, 15 Feb 2023 04:57:22 GMT
        	            				/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/testing.go:1150
Wed, 15 Feb 2023 04:57:22 GMT
        	            				/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/testing.go:1328
Wed, 15 Feb 2023 04:57:22 GMT
        	            				/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/testing.go:1570
Wed, 15 Feb 2023 04:57:22 GMT
        	Error:      	Received unexpected error:
Wed, 15 Feb 2023 04:57:22 GMT
        	            	context deadline exceeded
Wed, 15 Feb 2023 04:57:22 GMT
        	Test:       	TestDeployAllInOneDBLESS

Proposed Solution

• Add options to .Cleanup() to allow configurable behavior like waiting for namespaces to be deleted.

Additional information

No response

Acceptance Criteria

• As a user I can specify options for .Cleanup() and not wait for namespaces to be deleted.