-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Serialization failure, Deadlock found when trying to get lock #14757
Comments
Hmm I wonder if it's due to #14752 which introduces a transaction because that's the branch I got the issue on. |
@PVince81 Also @felixboehm mentioned that we maybe need to handle aborted transactions because of conflicts somehow. |
Hmmm just now I noticed that while I renamed "lots3" to "lots4", now I have two duplicate local folders called "lots3" and "lots4". Aborted transactions ? You mean when the sync client tries to do something but the server is still busy from the previous parallel operation ? |
No aborted transaction as in: The database says "no I can't do this transaction, because there was a parallel concurrent transaction and this transaction isn't valid anymore" |
I tried to reproduce the issue but this run didn't. One thing I observed though is that the sync client sends a MOVE operation for EVERY file inside the renamed folder instead of just renaming the folder. It doesn't seem to always do that though. |
I'm trying to sequentially rename "lotsX" to "lotsY" and see what happens, in the hope to reproduce the issue.
(notice that lots9 is a folder, not a file) |
I tried several times and didn't manage to reproduce the deadlock issue. |
We've trying OC 8.1.0.6 beta2 on a cloud (HaProxy on frontend apache, HaProxy on frontend MariaDB Cluster) and there is the same problem. Not due to renaming directory, so not due to #14752 cause we've the last version where it's resolved, and we caused the problem without renaming directory. |
hi all, we have the same problem with OC8.0.5 and client version 1.8.4 ( build 2531 ), when we upload big files ( eg 300MB), the synchronization will abort with "Internal server error" Jul 17 09:29:02 sdcoca15 ownCloud[24989]: {remote} An exception occurred while executing 'UPDATE Probably is the same bug? |
Is anyone still seeing this with 8.1.3 with transactional redis locking enabled ? (https://doc.owncloud.org/server/8.1/admin_manual/configuration_files/files_locking_transactional.html) |
We are using OC 8.1.2 with transactional filelocking and MariaDB and are still getting this error. |
Can you check the "Admin" page and under "Status" you should see a message that says that locking is enabled. |
The message is visible. Also our behaviour is, that after the message the sync client does not sync the new files. I think this is because of the unchanged etag |
Hmmm, the table "file_locks" doesn't exist in 8.1.x. The app "files_locking" is something different and doesn't use the database. |
I just upgraded to OC 8.2.0 and got the same error again. The updated file is shown in the web interface but no client downloads the changes because the etag of the parentfolder is not changed. The The exact error message: {"reqId":"DEPld/dVX2BKKGWwRuGK","remoteAddr":"137.250.18.11","app":"index","message":"Exception: {"Exception":"Doctrine\DBAL\Exception\DriverException","Message":"An exception occurred while executing 'UPDATE |
#20555 refers to the |
Actually with the new locking this should not happen any more. But if you observed it on 8.2.0 where the new locking exists, then there might be an additional issue. |
Hi, I add a file to sync folder which is a big file about 4Gb. Then every sync operation gives Jul 17 09:29:02 sdcoca15 ownCloud[24989]: {remote} An exception occurred while executing 'UPDATE oc_filecache SET mtime = ?, etag=? WHERE (mtime <> ? OR etag <> ? ) AND fileid = ? ' with params [1437118142, "55a8aebee9f3a", 1437118142, "55a8aebee9f3a", "11251183"]:#12#012SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction error. i am not using redis cache, only php cache... |
Please update to 8.2.2, it solves a few locking issues when using DB locking (like stray locks) @catborise you seem to have a different issue, so you say it always happen when hitting insufficent space ? Maybe that code path somehow doesn't free the database table ? |
Hello everybody,
|
Some cleanup background jobs can also cause the DB to be locked for too long: #22243 |
8.2.2: https://github.com/owncloud/enterprise/issues/1150 |
@PVince81 @icewind1991 I could have a look at this instance - what are we looking for? What information is needed to debug this? |
@MorrisJobke not sure. It might depend for what query it happens, then need to find out why it happens in the first place. |
Looking at @michi-roth's stack trace here #14757 (comment) it looks like the file was modified outside of ownCloud, and at the time of the PROPFIND ownCloud noticed the change and tried to update the cache and propagate it. @michi-roth did you explicitly modify files outside of ownCloud, directly in the data folder ? If not, then this might be a side-effect of another issue. |
So far there are only to possibly long-running transactions:
Normally as I said before, the new file locking from 8.2/9.0 should be able to prevent the DB code to be reached in the first place. Let's see whether there are some code paths that slip through the cracks and still produce this condition. |
I found this fix 26939a2 but it seems that it was already in 9.0 RC1, where the report has been reported again. |
@michi-roth can you check whether you have a setting "file_system_check_changes" in your config.php ? Your last stack trace seems to imply that you have explicitly set this value to 1, while the default is 0. Usually setting it to 0 would solve the issue. |
For now I'll assume that you had "filesystem_check_changes" set to 1. Or maybe it was an external storage for which update detection is enabled anyway, so the situation is likely to happen there too. Here is my first test run:
The sleep call will make sure that every entry in the folder takes 10 to be renamed in the database. This test did NOT reproduce the issue and showed that locking worked properly so far. |
Side note: the web UI's regular "getstoragestats.php" call will also trigger a scan while trying to computer the used space. That one also successfully was excluded thanks to the lock. |
Next step: try the reverse condition: have the scanner run first and then try and rename. |
So far I could not reproduce this exact issues. All my attempts at reproducing it lead to a deadlock on oc_file_lock, but not oc_filecache, which were documented here: #20555 (comment) |
I'm out of ideas for this and was unable to reproduce the deadlock on oc_filecache. Moving to 9.0.3 to keep in scope, and hopefully you guys will have upgraded to 9.0.2 and reported back whether the fix worked. Thanks! |
Is anyone still seeing this in 9.0.2 ? |
We were able to find the reason for the error in our configuration. The problem was the MariaDB Galera cluster when write requests were sent to all servers deadlock can occur. The solution was to send all write requests to a single server. |
Verified ! No error since I changed my HAproxy configuration, thank you ! |
@michi-roth thanks a lot for the hint. I raised a ticket for the documentation, maybe having it as a hint there would help other people with this issue: owncloud-archive/documentation#2490 |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Steps to reproduce
Expected result
Renamed properly.
Actual result
Very strange internal server error:
And then the sync client reuploads all the files which takes a while...
Versions
ownCloud 8.0.0 / stable8 found while testing #14752
mariadb-10.0.14-2.3.x86_64
php5-5.6.6-1.1.x86_64
openSUSE Factory
Not sure if this can be reproduced consistently, will try again.
That could be another argument for closeCursor.
@DeepDiver1975 @icewind1991 @MorrisJobke
The text was updated successfully, but these errors were encountered: