Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OrientDB 2.0~2.0.2: deadlocked by KEEP_EXCLUSIVE_LOCK #3593

Closed
vtliem opened this issue Feb 16, 2015 · 2 comments
Closed

OrientDB 2.0~2.0.2: deadlocked by KEEP_EXCLUSIVE_LOCK #3593

vtliem opened this issue Feb 16, 2015 · 2 comments
Milestone
2.1-rc3

Comments

@vtliem
Copy link

vtliem commented Feb 16, 2015

1 Init a DB pool:

mDBPool = new OPartitionedDatabasePool(url, user, pwd,
            OGlobalConfiguration.DB_POOL_MAX.getValueAsInteger());

*plocal,embedded, DB was upgraded from orientDB 1.7.10 and a fresh db created with orientdb 2.0.2
2 Get a db

db = mDBPool.acquire();

3 Update a record:

ODatabaseDocumentTx adb = mDBPool.acquire();
try{
    ODatabaseDocumentTx db = adb;
   try{
         db.begin();
         ODocument doc =db.load(id, null, false, false,
            KEEP_EXCLUSIVE_LOCK);
         //update fields and delete linked records,update index
         doc.save();
         db.commit();
         db = null;
    }finally{
        if(db !=null)db.rollback();
    }
}finally{
        adb.close();
}

4 Second time update, got a OTimeoutException: Can not lock record for 2000 ms. seems record is deadlocked by other record

  1. Log
removeUser:"data":{"sid":"#19:311","user":"#10:77"}
2015-02-16 10:35:57.845:DBUG:jcsscu.DBUtils:qtp1499825229-34: Open:com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPolled@12eef3b2
2015-02-16 10:35:57.859:DBUG:jcsscm.GroupChatModel:qtp1499825229-34: remove user gusers#20:2817{sid:#19:311,user:#10:77,join:1424049549502,sent:1424049773179,joinno:60,no:0} v3
2015-02-16 10:35:58.025:DBUG:jcsscu.DBUtils:qtp1499825229-34: Close:com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPolled@12eef3b2

removeUser:data":{"sid":"#19:311","user":"#10:12"}
2015-02-16 10:36:04.853:DBUG:jcsscu.DBUtils:qtp1499825229-37: Open:com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPolled@51571f16
2015-02-16 10:36:06.858:WARN:jcsscu.DBUtils:qtp1499825229-37: timeout:KEEP_EXCLUSIVE_LOCK
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #19:311 (cluster: gchatsession)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1605)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:226)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1437)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:210)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:261)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:277)
    at jp.co.siy.siychat.chatsvr.model.GroupChatModel.removeUser(GroupChatModel.java:1580)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:64)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:160)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:613)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:468)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: 
com.orientechnologies.common.concur.OTimeoutException: Can not lock record for 2000 ms. seems record is deadlocked by other record
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.acquireWriteLock(OAbstractPaginatedStorage.java:1288)
    at com.orientechnologies.orient.core.tx.OTransactionAbstract.lockRecord(OTransactionAbstract.java:118)
    at com.orientechnologies.orient.core.id.ORecordId.lock(ORecordId.java:282)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.lockRecord(OAbstractPaginatedStorage.java:1790)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1424)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:697)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1572)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:226)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1437)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:210)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:261)
    at jp.co.siy.siychat.chatsvr.utils.DBUtils.load(DBUtils.java:277)
    at jp.co.siy.siychat.chatsvr.model.GroupChatModel.removeUser(GroupChatModel.java:1580)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:64)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:160)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:613)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:468)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
2015-02-16 10:36:06.859:DBUG:jcsscu.DBUtils:qtp1499825229-37: Close:com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPolled@51571f16

Note:
1.This code run without issues in OrientDB 1.7.
2.At close(ODatabaseDocumentTx db)

db.getTransaction()
            .getLockedRecords() returns empty map.
@vtliem
Copy link
Author

vtliem commented Feb 16, 2015

I found the issue, in removeUser funtions, rid.lock(xxx) is multiple called (by db.load and by code) and rid.lock(xxx) doesn't check current locking state. How to rid.lock only new LOCKING_STRATEGY >current LOCKING_STRATEGY?
Now, in load record, I do:

if (lockingStrategy != LOCKING_STRATEGY.NONE) {
        LOCKING_STRATEGY currentLock = db.getTransaction()
            .getLockedRecords().get(rid.getIdentity());
        if (currentLock != null) {
        if (currentLock.compareTo(lockingStrategy) >= 0) {
            LOG.debug(
                "multiple lock:{}:current:{},new:{} and current lock >= new lock: set new lock to NONE",
                rid.getIdentity(), currentLock, lockingStrategy);
            lockingStrategy = LOCKING_STRATEGY.NONE;
        } else {
            LOG.debug("multiple lock:{}:current:{},new:{} and current lock < new lock:unlock current",rid.getIdentity(), currentLock, lockingStrategy);
            rid.unlock();
        }
        }
    }

but I'm not sure it's ok and don't know how to do this in

new ORecordIteratorClass<ODocument>(final ODatabaseDocumentInternal iDatabase, final ODatabaseDocumentInternal iLowLevelDatabase,
      final String iClassName, final boolean iPolymorphic, final boolean iUseCache, final boolean iterateThroughTombstones,
      final OStorage.LOCKING_STRATEGY iLockingStrategy)

@vtliem vtliem changed the title OrientDB 2.0~2.0.3: deadlocked by KEEP_EXCLUSIVE_LOCK OrientDB 2.0~2.0.2: deadlocked by KEEP_EXCLUSIVE_LOCK Feb 16, 2015
@vtliem vtliem mentioned this issue Feb 25, 2015
Closed
@andrii0lomakin
Copy link
Member

You should not use KEEP_EXCLUSIVE lock strategy instead you should load record, lock it by call of record.lock method (it is possibly only in non tx mode and for embedded database) and then call record.reload(null, true, false) to have latest up to date version of your data. Reload will happen only if your version and version stored in storage are different

@andrii0lomakin andrii0lomakin added this to the 2.1-rc3 milestone May 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
2.1-rc3
Development

No branches or pull requests
2 participants
@andrii0lomakin @vtliem