[BUG] Cluster scale up - one pod not joining the cluster

[aaronfern]

Describe the bug:
When an etcd cluster is scaled up, there is a rare case where the third pod cannot join the cluster. Hence, we have only 2 members working is expected ways. What this results in is a cluster is size 2 with a learner that cannot join.
Although this means that the quorum is maintained, it also means that we lose a bit of resilience as one we are now just pod away from quorum loss.

The third pod complains of a cluster ID mismatch error and hence refuses to join the cluster. A possible cause of this is incorrect configuration passed to the etcd container before it is ready to join the cluster. The root cause however, is not very clear.

Expected behavior:
Cluster scale up to always succeed

How To Reproduce (as minimally and precisely as possible):
It is not yet clear how to consistently reproduce this. It is seen quite sporadically.

[abdasgupta]

Closing this issue as the problem is resolved