Getting exceptions detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached and transactions to database stoped. #4465

shlapkov · 2015-06-25T21:50:51Z

Getting exception when running in distributed mode. Steps to reproduce: 1. Start Node#1, Start Node#2. 2. Stop Node#2 Node#1 is still processing transactions. 3. Start Node#2 again. Notice errors in the log and all data lost. WARN [ODistributedResponseManager] [node_vmlnxstp2_8080] detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached, rolling back changes for request (id=342 from=node_vmlnxstp2_8080 task=tx userName=admin)
WARN [ODistributedResponseManager] [node_vmlnxstp2_8080] Quorum 2 not reached for request (id=342 from=node_vmlnxstp2_8080 task=tx userName=admin). Elapsed=62ms Servers in timeout/conflict are:

node_vmlnxstp2_8080: [Native support for inheritance [moved] #10:116 v.2]
Received: {node_vmlnxstp2_8080=[Native support for inheritance [moved] #10:116 v.2], node_vmlnxstp1_8080=[Native support for inheritance [moved] #10:152 v.2]}
WARN [ODistributedResponseManager] [node_vmlnxstp2_8080] sending undo message for request (id=342 from=node_vmlnxstp2_8080 task=tx userName=admin) to server node_vmlnxstp2_8080
WARN [ODistributedResponseManager] [node_vmlnxstp2_8080] sending undo message for request (id=342 from=node_vmlnxstp2_8080 task=tx userName=admin) to server node_vmlnxstp1_8080

shlapkov · 2015-06-25T22:05:00Z

Issue easy to reproduce: Start stop on of the node and exception below: 2015/06/25 18:00:49 | Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 2 not reached for request (id=193 from=node_vmlnxstp1_8080 task=tx userName=admin). Elapsed=22ms Servers in timeout/conflict are:
015/06/25 18:00:49 | - node_vmlnxstp2_8080: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #105:0 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v19 your=v20)
2015/06/25 18:00:49 | Received: {node_vmlnxstp2_8080=com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #105:0 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v19 your=v20), node_vmlnxstp1_8080=[21, 24, 23, 23, 23]}
2015/06/25 18:00:49 | at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.manageConflicts(ODistributedResponseManager.java:585) ~[orientdb-server-2.0.11.jar:2.0.11]
2015/06/25 18:00:49 | at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.getFinalResponse(ODistributedResponseManager.java:349) ~[orientdb-server-2.0.11.jar:2.0.11]
2015/06/25 18:00:49 | at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.waitForResponse(OHazelcastDistributedDatabase.java:365) ~[orientdb-distributed-2.0.11.jar:2.0.11]
2015/06/25 18:00:49 | at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:192) ~[orientdb-distributed-2.0.11.jar:2.0.11]

shlapkov · 2015-06-25T22:43:07Z

I know the reason for it: Only after multiply restart issue is gone. Looks like orient 2.0.11 require all local cluster per class to be present. But we don't have control over this creations. Once I have all classes with 3 cluster for 2 nodes I don't see this issue any more and I can safely start/stop nodes in distributed mode without any issue. Please advice?

lvca · 2015-06-26T10:20:15Z

Could you post here the distributed*.json file in your database directory?

shlapkov · 2015-06-26T14:02:48Z

This file is constantly changes so posting different stages:

For first start with fresh database or after upgrade from: 1.7.x we are usin the following file: default-distributed-db-config.json

{
  "autoDeploy": true,
  "hotAlignment": false,
  "offlineMsgQueueSize" : 0,
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "clusters": {
    "internal": {
    },
    "index": {
    },
    "*": {
      "servers" : [ "<NEW_NODE>" ]
    }
  }
}

Starting node: vmlnxstp1 then vmlnxstp2. It is "broken" state. File from vmlnxstp1:

{"@type":"d","@version":0,"version":55,"autoDeploy":true,"hotAlignment":false,"offlineMsgQueueSize":0,"readQuorum":1,"writeQuorum":2,"failureAvailableNodesLessQuorum":false,"readYourWrites":true,"clusters":{"@type":"d","@version":0,"internal":{"@type":"d","@version":0},"index":{"@type":"d","@version":0},"*":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-servicetransactionsummary_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-accountpasswordpolicy_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webserviceparameter_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowactivity_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"orids_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobsnapshot_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriver_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"ofunction_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-reporttemplate_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-typemetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"orole_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-remoteserver_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobresourcebundle_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriverparameter_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-license_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowsnapshot_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobreport_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-managerproperty_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-aclmetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-expirationconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-aclmetadataoverride_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-fileserverconnection_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-rootdirectory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-sidmetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-activitystatus_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourceconnection_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobreportcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-serviceconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-objectidentitymetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-loggingconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-smtptransport_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"oschedule_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowreporttemplate_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webservice_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobhistory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-generaltask_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-useraccount_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-setting_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webserviceheader_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-separator_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowstage_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"ouser_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-productresourceconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowhistory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflow_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-folders_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflow_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-version_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-servicetransaction_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-versioncontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriverurl_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]}}}

After many restart reach the "normal" state. File from vmlnxstp1:

{"@type":"d","@version":0,"version":279,"autoDeploy":true,"hotAlignment":false,"offlineMsgQueueSize":0,"readQuorum":1,"writeQuorum":2,"failureAvailableNodesLessQuorum":false,"readYourWrites":true,"clusters":{"@type":"d","@version":0,"internal":{"@type":"d","@version":0},"index":{"@type":"d","@version":0},"*":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-dataflow_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriver_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-versioncontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webservice_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobreport_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-servicetransaction_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-sidmetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"orole_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowstage_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-objectidentitymetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflow_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourceconnection_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-serviceconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-managerproperty_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriverparameter_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-typemetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-loggingconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-reporttemplate_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-activitystatus_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-productresourceconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-setting_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-aclmetadataoverride_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-rootdirectory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowsnapshot_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webserviceparameter_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobresourcebundle_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-separator_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-license_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobsnapshot_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobhistory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-accountpasswordpolicy_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowreporttemplate_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"oschedule_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"orids_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-expirationconfiguration_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"ouser_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowactivity_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"ofunction_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-datasourcedriverurl_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-webserviceheader_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-folders_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-smtptransport_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-useraccount_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-processflowhistory_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-aclmetadata_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-servicetransactionsummary_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-jobreportcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-dataflowcontents_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-remoteserver_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-fileserverconnection_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-generaltask_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-version_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"_studio_node_vmlnxstp2_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp2_8080","node_vmlnxstp1_8080","<NEW_NODE>"]},"platform-useraccount_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-loggingconfiguration_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-typemetadata_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"ouser_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"orole_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-webserviceparameter_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-processflowhistory_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-processflowcontents_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-reporttemplate_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-servicetransaction_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-datasourcedriverurl_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-datasourcedriverparameter_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-expirationconfiguration_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-dataflow_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-jobsnapshot_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-webservice_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-activitystatus_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-objectidentitymetadata_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-processflowactivity_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-jobreport_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-datasourcedriver_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-processflowsnapshot_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-version_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"oschedule_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"_studio_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-versioncontents_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-productresourceconfiguration_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-jobhistory_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-dataflowcontents_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-accountpasswordpolicy_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-rootdirectory_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-servicetransactionsummary_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-jobreportcontents_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-webserviceheader_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-remoteserver_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-dataflowreporttemplate_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-dataflowstage_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-smtptransport_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-sidmetadata_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-generaltask_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-managerproperty_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-aclmetadata_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-license_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-folders_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-datasourceconnection_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"ofunction_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"orids_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-processflow_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-aclmetadataoverride_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-jobresourcebundle_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-fileserverconnection_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-setting_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-separator_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]},"platform-serviceconfiguration_node_vmlnxstp1_8080":{"@type":"d","@version":0,"servers":["node_vmlnxstp1_8080","node_vmlnxstp2_8080","<NEW_NODE>"]}}}

shlapkov · 2015-06-26T16:14:28Z

It maybe related to: http://www.prjhub.com/#/issues/4608

lvca · 2015-06-26T16:59:27Z

What's the broken state? Please can you point me on that in the JSON file?

jillray7 · 2015-06-26T17:02:55Z

Note from Rob: I just want to emphasize the importance of this issue. If we cannot get this resolved in 2.0.12, we will have to go back to the last known good build which is 2.0.7. That makes me nervous because if something is found in it, we are without an upgrade path other than getting to the latest.

shlapkov · 2015-06-26T17:43:56Z

File is not broken, But start stop node1 with running translations on the node2 may generate exceptions "1 node(s) in timeout or in conflict and quorum (2) has not been reached, rolling back changes for reque"s and after this only full restart for node#1, node#2 helps.

lvca · 2015-06-26T21:39:15Z

When node1 is down and node2 report correctly quorum not reached, is the node2 up & running or do you have any problem with it?

shlapkov · 2015-06-26T21:47:36Z

First start Node1 - create DB, Start Node 2 - all running fine. Stop Node 2 - transactions still processing on Node1 no issue. Start Node2 one of the nodes may starts generate exceptions. After this both needs to be stop/started

lvca · 2015-06-29T14:07:15Z

I'm looking at it.

shlapkov · 2015-06-30T15:32:35Z

Hello @lvca, Any updates on this? What is the plan for 2.0.12 release?

lvca · 2015-06-30T15:51:40Z

I'm still figuring out the reason behind this problem. I'd prefer to wait for 2.0.12 until this issue is fixed.

lvca · 2015-07-01T00:08:54Z

Found the problem and fixed. Waiting all tests pass.

lvca · 2015-07-01T00:20:25Z

Tested with your test case and 3 nodes with start and stop of each one, and everything is ok. I'm planning to put this use case in our suite (we have a similar one, but not the very same). In this way we can avoid regression on it.

lvca · 2015-07-01T00:21:27Z

This problem was due to an optimization in Hazelcast to read multiple msgs in one shot. On high pressure, the queue of unread messages could be drop during startup only.

shlapkov · 2015-07-01T01:15:37Z

@lvca Thank you! I'll check with 2.0.12 snapshot and will let you know about results

shlapkov · 2015-07-24T22:03:07Z

Notice the same exception when was trying to rebuild all indexes from studio in distributed mode:

jvm 1 | INFO [OIndexRebuildOutputListener] --> OK, indexed 1,042 items in 24 ms
jvm 1 | INFO [OIndexRebuildOutputListener] - Rebuilding index spectrum.platform-JobHistory2_idx
(estimated 89687 items)...
jvm 1 | INFO [OIndexRebuildOutputListener] --> OK, indexed 89,687 items in 3,590 ms
jvm 1 | INFO [OIndexRebuildOutputListener] - Rebuilding index spectrum.platform-JobSnapshot_pk (
estimated 89686 items)...
jvm 1 | INFO [OIndexRebuildOutputListener] --> 5.98% progress, 5,365 indexed so far (536 items/s
ec)
jvm 1 | WARN [OHazelcastDistributedDatabase] [node_VA005SH_AUS1_8080] timeout (60005ms) on waiti
ng for synchronous responses from nodes=[node_VA005SH_AUS1_8080] responsesSoFar=[] request=id=1383 f
rom=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userName=admin
jvm 1 | WARN [ODistributedResponseManager] [node_VA005SH_AUS1_8080] no response received from lo
cal node about request id=1383 from=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userNam
e=admin
jvm 1 | WARN [ODistributedResponseManager] [node_VA005SH_AUS1_8080] detected 1 node(s) in timeou
t or in conflict and quorum (1) has not been reached, rolling back changes for request (id=1383 from
=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userName=admin)
jvm 1 | WARN [ODistributedResponseManager] [node_VA005SH_AUS1_8080] Quorum 1 not reached for req
uest (id=1383 from=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userName=admin). Elapsed
=60006ms No server in conflict. Received: {node_VA005SH_AUS1_8080=waiting-for-response}
jvm 1 | ERROR [ONetworkProtocolHttpDb] Internal server error:
jvm 1 | com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing
distributed request (id=1383 from=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userName
=admin) against database 'spectrum.[]' to nodes [node_VA005SH_AUS1_8080]
jvm 1 | --> com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 1 not r
eached for request (id=1383 from=node_VA005SH_AUS1_8080 task=command_sql(REBUILD INDEX *) userName=a
dmin). Elapsed=60006ms No server in conflict. Received: {node_VA005SH_AUS1_8080=waiting-for-response
}
jvm 1 | INFO [OIndexRebuildOutputListener] --> 10.28% progress, 9,217 indexed so far (385 items/
sec)
jvm 1 | INFO [OIndexRebuildOutputListener] --> 14.58% progress, 13,077 indexed so far (386 items
/sec)
jvm 1 | INFO [OIndexRebuildOutputListener] --> 16.42% progress, 14,729 indexed so far (165 items
/sec)
jvm 1 | INFO [OIndexRebuildOutputListener] --> 17.70% progress, 15,873 indexed so far (114 items
/sec)
jvm 1 | INFO [OIndexRebuildOutputListener] --> 19.88% progress, 17,829 indexed so far (195 items

lvca · 2015-07-30T17:55:24Z

This could be the same as for #2914 just fixed. Tried with a fresh database after the fix for #2914 and worked. Please could you retry with last 2.0.x branch?

lvca · 2015-07-30T18:15:19Z

I'm trying with your database you sent via email to see if it's fixed.

shlapkov · 2015-07-30T18:15:35Z

I can't find any #2914

lvca · 2015-07-30T18:16:48Z

#2914

lvca · 2015-07-31T08:39:49Z

@tglman tried with your database and works. Any updates on your side about this?

jillray7 · 2015-07-31T16:24:06Z

@shlapkov Please retry with last 2.0.x branch and let us know if the issue is resolved.

shlapkov · 2015-07-31T19:01:52Z

Issue not fixed, Please have a look on the test: https://github.com/shlapkov/OrientIssues/blob/master/BrokenReplications/OrientDBCluster.java

First run with pom against 2.0.7: https://github.com/shlapkov/OrientIssues/blob/master/BrokenReplications/pom.xml

<!--
    <orientdb.version>2.0.14-SNAPSHOT</orientdb.version>
    -->
    <orientdb.version>2.0.7</orientdb.version>
then run with:
    <orientdb.version>2.0.14-SNAPSHOT</orientdb.version>

After start stop start europe1 Notice

Received: {europe0=[#17:792 v.291], europe1=[#17:653 v.285]} [ODistributedResponseManager]
2015-07-31 13:42:53:221 WARNING [europe1] sending undo message for request (id=3507 from=europe1 task=tx userName=admin) to server europe0 [ODistributedResponseManager]
2015-07-31 13:42:53:221 WARNING [europe1] sending undo message for request (id=3507 from=europe1 task=tx userName=admin) to server europe1 [ODistributedResponseManager]
2015-07-31 13:42:53:222 SEVERE ******************************** [OrientDBCluster]Create Iteration=4, id=ID-773ee87a-3b0f-41ac-8ae3-41983d21b0e8, com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=3507 from=europe1 task=tx userName=admin) against database 'platform.[person_europe1]' to nodes [europe0, europe1]
com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=3507 from=europe1 task=tx userName=admin) against database 'platform.[person_europe1]' to nodes [europe0, europe1]
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:195)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:354)
    at com.orientechnologies.orient.server.distributed.ODistributedStorage.commit(ODistributedStorage.java:923)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:488)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:147)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2437)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2407)
    at test.OrientDBCluster$Reader.createRecord(OrientDBCluster.java:640)
    at test.OrientDBCluster$Reader.call(OrientDBCluster.java:563)
    at test.OrientDBCluster$Reader.call(OrientDBCluster.java:534)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 2 not reached for request (id=3507 from=europe1 task=tx userName=admin). Elapsed=3ms Servers in timeout/conflict are:
 - europe1: [#17:653 v.285]
Received: {europe0=[#17:792 v.291], europe1=[#17:653 v.285]}
    at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.manageConflicts(ODistributedResponseManager.java:585)
    at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.getFinalResponse(ODistributedResponseManager.java:349)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.waitForResponse(OHazelcastDistributedDatabase.java:358)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:192)
    ... 13 more

2015-07-31 13:42:53:222 SEVERE ******************************** [OrientDBCluster]

jillray7 · 2015-08-03T05:57:37Z

@lvca Can you update this issue? Did you fix it in the latest 2.0.x branch?

lvca · 2015-08-03T07:11:46Z

This is fixed in 2.0.x and develop branch. Please @shlapkov can you confirm it?

shlapkov · 2015-08-03T17:59:13Z

Still see the issue as well some other issues with today local build: 2.0.14-SNAPSHOT:
2015-08-03 09:54:28:974 SEVERE ******************************** [OrientDBCluster]
2015-08-03 09:54:28:974 SEVERE ******************************** [OrientDBCluster]Create Iteration=1, id=plocal:D:/SVN/dev/tools/orientdb-cluster-test/servers/1/databases/platform.44-28b6a3ba-25c5-4665-9e74-14f0383ac499, com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=2057 from=europe1 task=tx userName=admin) against database 'platform.[person_europe1]' to nodes [europe0, europe1]
com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=2057 from=europe1 task=tx userName=admin) against database 'platform.[person_europe1]' to nodes [europe0, europe1]
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:195)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:354)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.commit(ODistributedStorage.java:923)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:488)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:147)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2437)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2407)
at test.OrientDBCluster$Writer.createRecord(OrientDBCluster.java:360)
at test.OrientDBCluster$Writer.call(OrientDBCluster.java:207)
at test.OrientDBCluster$Writer.call(OrientDBCluster.java:189)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 2 not reached for request (id=2057 from=europe1 task=tx userName=admin). Elapsed=44ms Servers in timeout/conflict are:

europe1: [Support browseClass and browseCluster from console tool [moved] #13:272 v.7]
Received: {europe0=[Support browseClass and browseCluster from console tool [moved] #13:524 v.10], europe1=[Support browseClass and browseCluster from console tool [moved] #13:272 v.7]}
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.manageConflicts(ODistributedResponseManager.java:585)
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.getFinalResponse(ODistributedResponseManager.java:349)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.waitForResponse(OHazelcastDistributedDatabase.java:358)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:192)
... 13 more

lvca · 2015-08-04T08:49:03Z

This seems the same error as before. Did you build 2.0.x branch manually or did you just get the last snapshot from sonatype? We push builds on sonatype repositories regularly but sometimes the process takes more than 24h.

jillray7 · 2015-08-04T14:17:04Z

@shlapkov Do you have a comment for the note above?

shlapkov · 2015-08-04T15:54:03Z

Yes I'll try both manual build and sonatype will let you know about results.

shlapkov · 2015-08-04T20:07:40Z

Hello @lvca I am having difficulties with today build out of 2.0.x see #4907. However before I updated branch I notice the same exception when reproduced ticket #4608 sent you email

jillray7 · 2015-08-05T18:30:53Z

@shlapkov Can you let us know if this is resolved, now that 4907 is closed?

shlapkov · 2015-08-05T19:29:09Z

Hi @jillray7, @lvca yes looks like with latest build out of 2.0.x after running OrientDBCluster several times I am not able recreate. However, I am not sure if it is closed because still see similar exception with new test app OrientDBClusterReplication for #4608 but it maybe be something different. We probable can can close this ticket if somebody still investigating issue #4608.

jillray7 · 2015-08-05T22:10:44Z

@shlapkov Great news! @lvca Let's see if we can knock out #4608.

prjhub added the bug label Jun 25, 2015

lvca added the waiting reply label Jun 26, 2015

lvca self-assigned this Jun 26, 2015

prjhub removed the waiting reply label Jun 26, 2015

lvca added the in progress label Jun 26, 2015

lvca added the waiting reply label Jun 26, 2015

lvca added the in progress label Jun 29, 2015

lvca added the waiting reply label Jun 30, 2015

lvca added a commit that referenced this issue Jul 1, 2015

Fixed issue #4465

3f67832

lvca added a commit that referenced this issue Jul 1, 2015

Fixed issue #4465

e181c55

Getting exceptions detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached and transactions to database stoped. #4465

Getting exceptions detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached and transactions to database stoped. #4465

Comments

shlapkov commented Jun 25, 2015

shlapkov commented Jun 25, 2015

shlapkov commented Jun 25, 2015

lvca commented Jun 26, 2015

shlapkov commented Jun 26, 2015

shlapkov commented Jun 26, 2015

lvca commented Jun 26, 2015

jillray7 commented Jun 26, 2015

shlapkov commented Jun 26, 2015

lvca commented Jun 26, 2015

shlapkov commented Jun 26, 2015

lvca commented Jun 29, 2015

shlapkov commented Jun 30, 2015

lvca commented Jun 30, 2015

lvca commented Jul 1, 2015

lvca commented Jul 1, 2015

lvca commented Jul 1, 2015

shlapkov commented Jul 1, 2015

shlapkov commented Jul 24, 2015

lvca commented Jul 30, 2015

lvca commented Jul 30, 2015

shlapkov commented Jul 30, 2015

lvca commented Jul 30, 2015

lvca commented Jul 31, 2015

jillray7 commented Jul 31, 2015

shlapkov commented Jul 31, 2015

jillray7 commented Aug 3, 2015

lvca commented Aug 3, 2015

shlapkov commented Aug 3, 2015

lvca commented Aug 4, 2015

jillray7 commented Aug 4, 2015

shlapkov commented Aug 4, 2015

shlapkov commented Aug 4, 2015

jillray7 commented Aug 5, 2015

shlapkov commented Aug 5, 2015

jillray7 commented Aug 5, 2015