OrientDB concurrent writes in distributed mode - not working

[arpitaagarwal]

OrientDB Version: 3.0.10

Java Version: 8

OS: MacOS 10.13.6 and Ubuntu 16.04.3 LTS (Its a containerised environment)

Expected behavior

We have a write intensive application and need to deploy OrientDB in distributed mode for HA in production. We are using OrientDB blueprint APIs.
OrientDB client should be able to connect to the cluster and create/write edges and vertices concurrently distributing load across different nodes. This is required for -
(1) Faster writes
(2) High Availability

Actual behavior

2 nodes cluster works fine with 1-2 threads. As number of threads increases, it starts giving timeout exceptions and OrientDB client hangs forever waiting for synchronisation to complete. It never recovers.

Steps to reproduce

1. A default deployment of OrientDB 3.0.10 without any config changes.
2. Start 2 nodes on orientdb through dserver.sh file on same machine. Both these nodes will start communicating with each other by sharing ip-address through multicast.
3. Use orientdb client connects to server with ipaddress, no stick ports.
4. Client writes a vertex-edge-vertex (Person-FriendOf-Person) through a single commit operation.
5. Add unique index on Person.name to avoid creating multiple vertex of same name.
6. Run this test multiple time with single thread (works fine).
7. Increase number of threads to see following in logs - Timeout (1244ms) on waiting for synchronous responses from nodes=[node2, node1] responsesSoFar=[node2] request=(id=1.105226 task=TxPhase1) [ODistributedDatabaseImpl]

Can someone suggest what's wrong in this deployment? We need to write millions of vertex-edge-vertex combinations and build a graph.
cc @lvca @tglman @Andrey Lomakin @luigidellaquila

[cschockaert]

+1 got exactly the same problem with odb 3.3.18 here
never got this problem in odb latest 2.X version

[Jotschi]

@tglman Do you think this issue is similar to the ones that were handled in private issue (10586 - Cluster communication failure) - Is this solved with 3.0.19?

[Jotschi]

Relates to #8742