Investigate possible etcd persistence issues

[SandeepPissay]

During design discussions with @luomiao, we discussed that etcd will lose all its data if all the manager nodes are offline for some time. This means that all the vFile related data we store in it are gone even after the etcd cluster is reestablished. We want to validate that since this is important from vFile design perspective. So filing this issue.

Possible persistence issue with etcd:
(scenario 1)

• Deploy a etcd cluster with 3 or more nodes. Each etcd node will have a data volume attached where it stores it's state.
• Add some data
• Power off all the nodes where etcd is running
• Bring it back.
• Validate if any data is lost.

(scenario 2)

• Deploy a etcd cluster with 3 nodes. Each etcd node will have a data volume attached where it stores it's state.
• Add some data
• Power off node 1 and node 2.
• Add/modify data in node3.
• Power off node 3.
• Power on node 1 and node 2.
• Add/modify data in node1 or node2.
• Power on node 3
• Validate if the data is as expected, does data gets merged etc.

Based on the results found, we can discuss if the etcd behavior is acceptable for vFile.

We discussed that we want to try a similar issue with Docker swarm as well. The manual test procedure we wanted to follow are:

• Deploy docker swarm with 3 managers and some workers.
• Deploy a service/app on it. Check if the service is listed
• Power off all manager nodes and bring them back.
• Check if the service is still listed to verify that the docker swarm did not lose data.

[SandeepPissay]

By quickly glancing through the etcd documentation, the above issues should not exist. @luomiao can you take this as you had some doubts?

[luomiao]

Closed by #1945