Fix status update in case of connection errors

[aledbf]

What this PR does / why we need it:

In case of any error (like the API server not being temporarily available) updating the ingress status, the leader election code stops working. This change forces the termination of the running elector, starting a new one.

Which issue this PR fixes:

fixes #3180
fixes #3033

Test image: quay.io/aledbf/nginx-ingress-controller:0.415

To test this we need to simulate network issues:

1. In a terminal run kubectl proxy --address=0.0.0.0 --accept-hosts=.*
2. In a different terminal:

export HOST_IP=<IP eth0 or ne0>

docker run --net=host -it \
    -e POD_NAMESPACE=ingress-nginx \
    -e POD_NAME=local \
    quay.io/aledbf/nginx-ingress-controller:0.415 \
    /nginx-ingress-controller \
    --configmap=ingress-nginx/nginx-configuration \
    --apiserver-host=http://$HOST_IP:8001 \
    --publish-status-address=1.1.1.1

3. wait a minute and terminate kubectl proxy
4. wait a minute and start kubectl proxy again

the election of a new leader should appear in the log after the reconnection

[ElvinEfendi]

No regression test?

[aledbf]

No regression test?

Writing one :)

[codecov-io]

Codecov Report

Merging #3267 into master will decrease coverage by 0.04%.
The diff coverage is 65.45%.

@@            Coverage Diff             @@
##           master    #3267      +/-   ##
==========================================
- Coverage   48.36%   48.31%   -0.05%     
==========================================
  Files          75       75              
  Lines        5554     5561       +7     
==========================================
+ Hits         2686     2687       +1     
- Misses       2523     2529       +6     
  Partials      345      345

Impacted Files	Coverage Δ
internal/ingress/controller/nginx.go	15.71% <0%> (ø)	⬆️
internal/ingress/status/status.go	71.42% <66.66%> (-1.65%)	⬇️
internal/watch/file_watcher.go	80.76% <0%> (-3.85%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc6f2e7...8522c80. Read the comment docs.

[aledbf]

@ElvinEfendi this is now ready with e2e test 😉
Log of e2e test pod https://gist.github.com/aledbf/c8670927ef144c0523bf7fd763f5c2c2

[aledbf]

This block of code was moved here to avoid running unnecessary goroutines without an actual update to run

[ElvinEfendi]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, ElvinEfendi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ElvinEfendi,aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ElvinEfendi]

thanks for the regression test ❤️