Cannot unlock a never acquired lock

[martingg88]

The following transaction caused "Cannot unlock a never acquired lock".
i guess the second let query (etc: user) can't retrieve account data as the first let query (etc: account) still lock the account class. any idea how to solve this issue?

BEGIN
LET account = SELECT * FROM account WHERE email = "cl@hotmail.com" LIMIT 1
LET user = SELECT EXPAND(in("own")) FROM account WHERE email = "cl@hotmail.com"
LET payment = SELECT EXPAND(out("own")[@Class =payment]) FROM $user
COMMIT

[andrii0lomakin]

Hi,
Could you provide DB schema ?
Is it correct that we merely need to run this query in console and we will reproduce this exception ?

[martingg88]

seem like it work again for orientdb-m3 version. what is the different in between orientdb-m3 and orientdb-2.0.3

[andrii0lomakin]

@martingg88 they are different in thousands lines of codes. Could you answer on my questions above ?

[martingg88]

i'm sorry about that. it's hard to catch this issue as it is not happen oftenly. will update you once i have this issue again.

[martingg88]

@Laa... this happen again in turn cause error "Error on retrieving record". this acquired lock will happen if i run the program for few times. FYI, the program only execute the following sql and then caused this issue. I bet this is only happen over http protocol.

SELECT EXPAND(out("follow")[@Class ="interest"]) FROM #23:28.

here is the simple database schema that caused that issue.

user follow interest

user

@Rid name
#23:28 ck

follow

ctime mtime
2015-02-28T09:09:58+0000 2015-02-28T09:09:58+0000

interest

@Rid title
#33:38 sport

[andrii0lomakin]

Emanuele could you look at this ?

[martingg88]

any update on this?
here is the details of error.

{db=sport} Error on fetching record during browsing. The record has been skipped
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #35:508 (cluster: lottery)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1604)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:80)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1437)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:117)
at com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:278)
at com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:147)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1300)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:433)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:391)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:64)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1189)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1178)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:63)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1178)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:385)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:220)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)
Caused by: com.orientechnologies.common.concur.lock.OLockException: Cannot unlock a never acquired lock
at com.orientechnologies.orient.core.tx.OTransactionAbstract.unlockRecord(OTransactionAbstract.java:137)
at com.orientechnologies.orient.core.id.ORecordId.unlock(ORecordId.java:287)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.unlockRecord(OAbstractPaginatedStorage.java:1769)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1443)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:694)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1572)

[martingg88]

hi guys... we got a big issue here for our production line that will cause our web system down/corrupted in every one hour due to the orientdb lock/deadlock issue. i'm looking forward this issue can be solved soon. thanks.

[lvca]

What OrientDB version? Please always specify it.

[martingg88]

2.0.5 version

[tglman]

@martingg88 is this error happening in your case in distributed environment or standalone?

[martingg88]

it happen to both cases.

[martingg88]

fyi. you will have this issue if you are executing your query by using the node.js async parallel.

here is the async library..
https://github.com/caolan/async

[martingg88]

fyi to ease ur developing process.

if you try to run the following simple query with multiple times in async way and most likely you will get the lock issue.

1. select from class

[lvca]

This should be resolved in 2.0.7-SNAPSHOT and 2.1-SNAPSHOT. Please could you try it?

[martingg88]

sure..i will test it. let me know once you guys release 2.0.7-SNAPSHOT and 2.1-SNAPSHOT. thanks.

[lvca]

@martingg88 I'm going to release 2.0.7 and 2.1-rc1, but I'd like to have the confirm this issue is gone. Please could you try with 2.0.7-SNAPSHOT?

[martingg88]

sure. i will try it and keep you posted. thanks for your effort

[nsuneja]

Has this issue been confirmed to be resolved in 2.0.7 ?

[martingg88]

sorry. i have been busy on these few weeks. i will test it next week and let you know about the outcome. thanks.

[martingg88]

@lvca. I still facing this issue for version 2.0.8 even i have following setting

my website is totally dead once i hit this issue

[martingg88]

this issue is more critical part that will cause the server down. can we put the high priority to issue?

[lvca]

Increased priority to HIGH

[kowalot]

@martingg88 What's the reason to have only three selects in transaction without other data modification commands?
Do you use any record locking strategy in your system?
Imo.. your error in this place might be just an effect of not released lock in other UPDATE/INSERT operation. (OrientDB skipped the record in lottery cluster due to locking)
Please also if you can check this batch without COMMIT as there is no data modification commands.

[martingg88]

1. What's the reason to have only three selects in transaction without other data modification commands?

   The usage here is to combine results from different classes into one result/row

2. Do you use any record locking strategy in your system?
   I use locking strategy for insert/update operation only. not select operation.
   For this issue "OrientDB skipped the record in lottery cluster due to locking", i just use the simple
   select query, not batch involved.

3. Please also if you can check this batch without COMMIT as there is no data modification commands.
   Can i check the batch without commit statement?

FYI. You will have this issue if you run the following simple query multiple times by using the node.js async parallel. bear in mind it is just a simple query without batch anymore.
here is the async library..
https://github.com/caolan/async

a. insert into interest SET title = ?
b. UPDATE interest SET title = ? RETURN AFTER @Rid WHERE @Rid= ?

[martingg88]

any update on this issue?

[andrii0lomakin]

I am on it.

[martingg88]

many thanks for you effort

[andrii0lomakin]

Actually I fixed it. Funny issue. I will push code which you may try.

[martingg88]

what the funny about? can you share with me?

[martingg88]

keep me updated when you push the code, so i can test it out

[andrii0lomakin]

records reused during select, to decrease of gc overhead, but side effect is that identity of locked object was changed inside of lock manager so you would not able to unlock it.

[martingg88]

ic...hope the next release will make sure the lock/unlock can be work without any issue. if not.. my server will be crashing for lock issue. anyway...many thanks for your efforts.

[martingg88]

when you guys can release this ? we really need this urgently. if not my server will be crashing few times in a day and my client start making noise about this issue. hope we can have this issue solved sooner. Many thanks.

[nsuneja]

Hey Martin

Were you able to confirm the resolution of this issue in 2.1-rc2 ?

[martingg88]

i have reopen this case at #4090