Fix context issue during cleanup of kind clusters

[jainpulkit22]

Fix context issue during cleanup of kind clusters.

Fixes #6768.

[antoninbas]

this is not a good idea IMO. It feels like there can be a race condition where we delete the file while another job is holding the lock?

[jainpulkit22]

But untill we delete config.lock the other process will not be able to acquire the lock and it may panic.
Also since flock is not compatible with the Go based locking mechanism so what's you opinion on that should we use the alternative approach of writing the cluster names to a file and acquiring lock over that, or. should we move ahead with acquiring lock over the kubeconfig and use the below approach:

before invoking kind create cluster command we can use flock to wait for the other process to release the lock and then we should trigger this command so in that case flock will not interfere with the Go based locking mechanism its just that we will be introducing couple of unnecessary flocks in the code.

@antoninbas what's your opinion on this?

[antoninbas]

We should not use ~/.kube/config.lock, I think that's a given. We can pretend it doesn't exist.
So we need our own lock file. After that, we have 2 options, and you can choose which one you want to use:

1. have our own state file to keep track of cluster names and creation timestamps (which I was originally proposing)
2. only rely on kubectl / kind, and do not introduce our own state file. With this option we use flock to acquire a lock before calling kubectl / kind, as appropriate

I think both approaches will work

[jainpulkit22]

so for 2nd approach we again rely on ~/.kube/config.lock right? I think we can go ahead with this approach.

[antoninbas]

No we don't rely on ~/.kube/config.lock for either approach, as we have discussed about why this is not a viable option.

[jainpulkit22]

ideally we have it on all our testbeds so this should not be an issue @XinShuYang and @KMAnju-2021 can confirm.

[KMAnju-2021]

Yes, iirc we have this /var/lib/jenkins directory on all kind testbeds.

[antoninbas]

Actually it's good that you highlight this. /var/lib/jenkins/ is not an appropriate directory to use here as ci/kind/kind-setup.sh is used for local development, Github CI, etc. Please use a more "universal" directory, such as ~/.kube/antrea/ (you can create the directory if it doesn't exist). I guess ~/.antrea/ would also be a good choice if we don't want to write anything to the ~/.kube directory.

[jainpulkit22]

I guess ~/.antrea would be a better option here

[jainpulkit22]

@antoninbas can you have a look at the code changes, thanks.