Stopped reconciliation if quorumloss annotation is set by operator.

[abdasgupta]

How to categorize this PR?

/area TODO
/kind TODO

What this PR does / why we need it:
This PR enables ETCD controller to check for an annotation named druid.gardener.cloud/ignore-reconciliation in ETCD CR. The annotation is applied by human operator who wants to recover a cluster from permanent quorum loss. The ETCD controller does not reconcile until the annotation is removed.

Which issue(s) this PR fixes:
Fixes #
#437

Special notes for your reviewer:

Release note:

Applying the annotation `druid.gardener.cloud/ignore-reconciliation` on the ETCD CR will stop etcd-druid from reconciling it.
This is helpful for operators to apply any manual fixes to the ETCD components, such as manually fixing permanent quorum loss as per the [playbook](https://github.com/gardener/etcd-druid/blob/master/docs/operation/Recover_From_Etcd_Permanent_Quorum_Loss.md).

[gardener-robot]

@abdasgupta Labels area/todo, kind/todo do not exist.

[gardener-robot]

@abdasgupta Labels area/todo, kind/todo do not exist.

[aaronfern]

/assign

[aaronfern]

Thanks for the PR @abdasgupta! Looks good in general

One suggestion, consider using the annotation druid.gardener.cloud/ignore instead as it's more semantically accurate

[abdasgupta]

Thanks for the PR @abdasgupta! Looks good in general

One suggestion, consider using the annotation druid.gardener.cloud/ignore instead as it's more semantically accurate

Done

[aaronfern]

Thanks for the changes!
/lgtm

[ishan16696]

Please add an appropriate release note.

[shreyas-s-rao]

@abdasgupta thanks for the PR. Can you please add a test for this?
Also, please update the PR note and add a release note too.

[abdasgupta]

@abdasgupta thanks for the PR. Can you please add a test for this?
Also, please update the PR note and add a release note too.

@shreyas-s-rao Any idea how to add a test case for this? This PR is just not reconciling on addition of the annotation

[gardener-robot]

@abdasgupta Labels area/todo, kind/todo do not exist.

[gardener-robot]

@abdasgupta Labels area/todo, kind/todo do not exist.

[abdasgupta]

Updated the release note

[aaronfern]

Thanks for the changes @abdasgupta!

One nit and one suggestion from me

[ishan16696]

Please don't do a force-push it makes review difficult.
As we decided please address review as separate commit and at the end we will do merge-squash.

[shreyas-s-rao]

@abdasgupta Thanks for making the changes. Final few comments from my side.

Please also update the release note to add the full link to the document - https://github.com/gardener/etcd-druid/blob/master/docs/operation/Recover_From_Etcd_Permanent_Quorum_Loss.md

[gardener-robot]

@abdasgupta Labels area/todo, kind/todo do not exist.

[shreyas-s-rao]

@abdasgupta Thanks for making the requested changes.
/lgtm

[aaronfern]

/lgtm