Update to member lease annotation for peer URL TLS is now done by making a member API call

[unmarshall]

What this PR does / why we need it:
Update to member lease annotation for peer URL TLS is now done by making a member API call which accurately depicts the status of the enablement of change of peer URL TLS as seen by the etcd process. Previously only mounted configmap was looked at which is not sufficient as an additional restart of the etcd process is required for this to be reflected in the etcd member.

Which issue(s) this PR fixes:
Fixes Part of #712

Special notes for your reviewer:

Release note:

Update to member lease annotation for peer URL TLS is now done by making a member API call

[gardener-robot]

@unmarshall You need rebase this pull request with latest master branch. Please check.

[ishan16696]

/assign

[renormalize]

In an out of bands discussion it was mentioned that this PR would not be necessary for gardener/etcd-druid#777 to be merged, could this be elaborated further here for documenting why so? Please correct me if my understanding is wrong.

[unmarshall]

In an out of bands discussion it was mentioned that this PR would not be necessary for gardener/etcd-druid#777 to be merged, could this be elaborated further here for documenting why so? Please correct me if my understanding is wrong.

As per Update advertise peer URLs it was understood that one needs to call member API to update the peer URL and then restart the member for it come into effect. In the current code in master STS component first updates the STS mounting the peer secret volume (this causes a rolling update thus resulting in backup-restore taking the latest config map which contains the changed peer url). Once the backup-restore container restarts as part of initialization it makes the update member API call. Then druid deletes the pods to force a second restart. When working on gardener/etcd-druid#777 it was observed that in the case where we move from replicas 1->3 (with scheme of peer URL changes from http to https) then the second restart was not really required. Druid continues to patch the STS mounting the secret vol (preserving the original replica count of 1). This causes the single member to restart and while it restarts the embedded etcd already picks up the latest peer URL (with TLS enabled) and also correctly gets the tls resources that it then uses to start etcd. This effectively makes the explicit call to update the member URL (currently done in backup-restore) and force a second restart (done in druid) quite useless.

In another case where there is a 3 member cluster and now replicas are increased to 5 and TLS is enabled then one restart would no longer be sufficient. However in this case this PR will still not be useful.

[shreyas-s-rao]

/close since we don't need this PR for scale-up of etcds in gardener/etcd-druid#777