[Kubernetes] Release leader lock on shutdown for faster failover

[ChrsMark]

Follow up issue to keep track of know issue we hit when trying to release the leader lock on shut down:

Quoting @jsoriano who observed the issue

Killing the pod with the leadership seems to hang for some time, and it logs:

E1015 17:07:38.138511       1 leaderelection.go:296] Failed to release lock: Lease.coordination.k8s.io "metricbeat-cluster-leader" is invalid: spec.leaseDurationSeconds: Invalid value: 0: must be greater than 0
It doesn't seem serious because other pod correctly takes the leadership after some seconds.

I1015 17:07:54.577547       1 leaderelection.go:252] successfully acquired lease default/metricbeat-cluster-leader
But i wonder if the failover could be faster if the original leader succeeded releasing the lock.

Commented at #21896 (comment)
Related to kubernetes/client-go#762 kubernetes/client-go#754 kubernetes/kubernetes#85474

Update: should be fixed by kubernetes/kubernetes#80954

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

Options I see to solve this:

• Fork and contribute a fix in go-client. I think the problem is in the structure used to encode the JSON used for the lock release request, LeaseDurationSeconds is an int there, but it should be a pointer to int so it can be omitted when not needed. In the server side this value is optional.
• Try to send the request to release the lock using the REST API directly. This would avoid the need to maintain a fork. We could consider contributing a fix in any case.

[ChrsMark]

Should be fixed in upstream in 1.20 after kubernetes/kubernetes#80954.

[jsoriano]

Should be fixed in upstream in 1.20 after kubernetes/kubernetes#80954.

Great! Let's update go-client when a new version is available.