Proposal to support multi-node etcd clusters

[amshuman-kr]

What this PR does / why we need it:
Proposal to support multi-node etcd clusters.

Credits:
@abdasgupta @shreyas-s-rao for extensive discussions and content.
@swapnilgm for initial discussions and first draft of implementation.
@istvanballok for many interesting ideas, especially, tmpfs.
@vlerenc for many interesting inputs, including (but not limited to 🙂) ephemeral persistence and etcd container and backup-restore sidecar interaction.

Which issue(s) this PR fixes:

Special notes for your reviewer:
This is the proposal part of #107. The implementation will be done in further PRs.
Please find the human-friendly rendering of the document here (note: point in time).

Release note:

NONE

[abdasgupta]

LGTM

[dguendisch]

Very nice and thorough write up, thanks!
I have two questions:

1. On backup failure, is it required to fail the readiness probe, or could ectd as well be put in some read-only mode? Rationale would be to "only" fail writes, but still allow for reads.

2. Out of interest: do you have experience how fast leader fail-over performs on etcd? From own experience I know that e.g. MongoDB is super fast (new primary elected in <5secs after old primary breaks), while e.g. Elasticsearch is super slow (often up to one 1min). For Elasticsearch thus very careful tuning of readinessProbe, initialDelaySeconds, and so on is necessary if a rolling update should be somewhat HA. And I guess etcd will not write without a leader, i.e. during a new election?

[amshuman-kr]

• On backup failure, is it required to fail the readiness probe, or could ectd as well be put in some read-only mode? Rationale would be to "only" fail writes, but still allow for reads.

@dguendisch Very interesting idea. I hadn't thought of that. I just checked the etcd configuration and unfortunately, there doesn't seem to be any explicit configuration that can put etcd clusters into read-only mode short of losing quorum. Forcibly losing quorum on backup failure sounds too drastic.

Out of interest: do you have experience how fast leader fail-over performs on etcd? From own experience I know that e.g. MongoDB is super fast (new primary elected in <5secs after old primary breaks), while e.g. Elasticsearch is super slow (often up to one 1min). For Elasticsearch thus very careful tuning of readinessProbe, initialDelaySeconds, and so on is necessary if a rolling update should be somewhat HA. And I guess etcd will not write without a leader, i.e. during a new election?

I am afraid I do not have too much experience. Only from local playing around, I would say leader election is quite fast. The heartbeats are 100ms and default leader election timeout is 1000ms according to the documentation. https://etcd.io/docs/v3.4.0/tuning/#time-parameters. That is quite fast.

Also, during leader election the writes are queued instead of rejected. So, writes experience more latency at worst. They may timeout, of course. https://etcd.io/docs/v3.1.12/op-guide/failures/#leader-failure.

[vlerenc]

Awesome, thank you very much, very helpful. I have only a few nitpicks and proposals, but otherwise I found it very enlightening.

[amshuman-kr]

@vlerenc Thanks a lot for the suggestions! I have applied almost all of them 🙂

[amshuman-kr]

I have addressed the review comments so far. I will merge this PR now. If there are more concerns, please do raise an issue :-)