Deadlocks on bootstrapping in distributed settings

[xinhaoyuan]

Hi,

I'm testing the concurrency of swarm using my in-house tool (will be released soon!). I've found several deadlocks when bootstrapping swarm in distributed settings. Here is one potential sketch of how a deadlock could happen:

1. Initially node1, node2 were in cluster_wait state
2. node1 gets cluster_join, select to sync with node2, and enters syncing state
3. node2 puts sync of node1 into pending_sync_req
4. node2 gets cluster_join, select to sync with node1 (sync of node1 is still in pending!), and enter syncing state
5. node1 gets sync from node2 and discovered a tie, it decides to wait node2 for sync_reply
6. Because node1's sync to node2 is in pending, it will never get handled (and thus no sync_reply will be sent to node1)
7. Both nodes are waiting for each other. The syncing process becomes dead.

[xinhaoyuan]

I see a deadlock in my trivial test with three nodes [1], while it runs fine with two nodes.

Is there anything wrong with my test code?

[1] https://github.com/xinhaoyuan/morpheus-app-test/tree/master/swarm_test_simple

[farhadi]

I also have the same issue with 3 nodes.
I use a docker-compose file to start 3 containers at the same time and they all stuck syncing/waiting.

[arjan]

I think this issue is fixed by #118, please reopen if you still get it on master.

[xinhaoyuan]

It appears the same in my test case (with commit 1430212).

[farhadi]

I'm still having the same issue with latest master branch.

[beardedeagle]

reopening for visibility