Throw fatal error and ask user to run cleanup when a k8s resource already exists in the cluster

[reetasingh]

When k8s resource like Namespace, clusterrole, clusterrole, binding, service account already exists in the cluster the RunE2E() test fails because the Create() operation returns error. But the error in the logs are not clear. Throw Fatal error when resource already exists and ask to run cleanup

Eg: when namespace already exists in the cluster I am getting below error

reeta@Reetas-MacBook-Pro hydrophone % make run
go run main.go
12:20:53 INF API endpoint : https://127.0.0.1:57154
12:20:53 INF Server version : version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-19T15:42:59Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/arm64"}
12:20:53 INF Using conformance image : 'registry.k8s.io/conformance:v1.24.0'
12:20:53 INF Using busybox image : 'registry.k8s.io/e2e-test-images/busybox:1.36.1-1'
12:20:53 INF Test framework will start '1' threads and use verbosity '4'
12:20:53 INF namespace already exist e2e-conformance-test
12:20:53 INF namespace created 

12:20:53 ERR an empty namespace may not be set during creation
exit status 1
make: *** [run] Error 1

To recreate this failure run make run twice

With this fix in place, it works fine

reeta@Reetas-MacBook-Pro hydrophone % make run
go run main.go
14:04:22 INF API endpoint : https://127.0.0.1:50108
14:04:22 INF Server version : version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-19T15:42:59Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/arm64"}
14:04:22 INF Using conformance image : 'registry.k8s.io/conformance:v1.24.0'
14:04:22 INF Using busybox image : 'registry.k8s.io/e2e-test-images/busybox:1.36.1-1'
14:04:22 INF Test framework will start '1' threads and use verbosity '4'
14:04:22 ERR namespace already exist conformance. Please run cleanup to start a fresh run
exit status 1
make: *** [run] Error 1
reeta@Reetas-MacBook-Pro hydrophone % 

[reetasingh]

also, we were using wrong resource name in the log

[rjsadow]

This looks good, but how does it account for the state of an object? For example after cleanup is run then the namespace will still exist in a terminating state.

[reetasingh]

directory:
Executable path: /usr/local/bin/ginkgo
Args (comma-delimited): /usr/local/bin/ginkgo,--focus=[Conformance],--skip=,--noColor=true,/usr/local/bin/e2e.test,--,--disable-log-dump,--repo-root=/kubernetes,--provider=skeleton,--report-dir=/tmp/results,--kubeconfig=,%!s()
sep is " " args are "%!s()"split [%!s()]

This looks good, but how does it account for the state of an object? For example after cleanup is run then the namespace will still exist in a terminating state.

Thats true, we are not checking status of the resource here

[rakshitgondwal]

Can we also rework the logs here? The logs are somewhat misleading if a resource already exists:

12:24:55 INF namespace already exist conformance
12:24:55 INF namespace created conformance

[dims]

So, here's the thing, if the conformance namespace is already there then that means something that ran earlier will not be cleaned up. So we have to explicitly tell the user to clean the namespace by hand AND THEN re-run hydrophone.

If they don't and we try to reuse whatever exists, they will run into problems (tests may fail randomly etc)

So +1 to cleanup logs and be explicit about. BUT please don't let them continue when there is a namespace already that hydrophone did not create explicitly in the current run.

[dims]

/hold

[reetasingh]

So, here's the thing, if the conformance namespace is already there then that means something that ran earlier will not be cleaned up. So we have to explicitly tell the user to clean the namespace by hand AND THEN re-run hydrophone.

If they don't and we try to reuse whatever exists, they will run into problems (tests may fail randomly etc)

So +1 to cleanup logs and be explicit about. BUT please don't let them continue when there is a namespace already that hydrophone did not create explicitly in the current run.

Thanks @dims for the context. in that case we can just throw error if a resource already exists.

[reetasingh]

@dims @rjsadow I have fixed the PR to just throw fatal error if resource already exists

[rjsadow]

/lgtm
/hold for @dims review

[dims]

/approve
please feel free to cancel hold

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, reetasingh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[reetasingh]

/hold remove

[reetasingh]

/hold cancel