"Deadlock found when trying to get lock" in file locking #20555
Comments
|
I had to hack the (somewhat related to #20545) |
As per our docs we need READ_COMMITED SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;should fix the problem |
We are setting the transaction level in the ctor of the connection - what's going wrong here? |
core/lib/private/db/connection.php Line 135 in 705d208 |
maybe the account we use is not allowed to run this command? |
we explicit set the transaction level on each connection we open |
|
Try running <?php
require('lib/base.php');
$connection = \OC::$server->getDatabaseConnection();
$result = $connection->executeQuery('SELECT @@GLOBAL.tx_isolation, @@tx_isolation');
var_dump($result->fetchAll());To check that the transaction level is set correctly |
This is 5.5.45-MariaDB from OpenSUSE. |
|
Can run That should get info about the most recent deadlock |
|
Output below, note that we have restarted Apache and MySQL meanwhile. Let me know if we should add back the unmodified scan.php file to have newer results. |
Please do |
|
(is this 8.2.0 or soon-to-be-8.2.1?) |
|
There's no time left for 8.2.1 now, so 8.2.2 |
|
I mean what s3 is running |
Done. Will update as soon as it reappears.
This is 8.2.0 with #19976. |
|
Ok. It reappeared. I can easily trigger it by calling |
|
As per MySQL docs:
Same for MariaDB:
This is the same from mysql 5.7 down to 5.0, so not sure why our call which sets it for the session doesn't work: |
It does set the transaction level correctly for the session |
|
Ok going from the innodb status and mysql docs it seems to be caused by the Setting Another option would be to drop the auto incrementing column from the locks which isn't used atm and only added for "good database design" although removing that might come to bite us in the future. |
|
@PVince81 @DeepDiver1975 opinions on how to tackle the auto-increment issue Note that this issue might also show up in other places that use |
|
Yes, this is happening for only one User who is not then able to access her Home Folder (mounted with Windows Network Drive and $user variable). Error is:
Please have a look at entry 4230 in the log file at ~/github-issues/core/20555/ |
|
Another duplicate: #23043 (comment) 00005004 |
Galera cluster of MariaDB with Maxscale in front to have the writes only gone to the master and all reads distributed at the slaves. |
@MorrisJobke you mean this is the workaround ? Sounds similar to #20555 (comment) So it looks like this is something that can be solved by modifying the database/cluster configuration ? Anything else that needs to be done on the code side ? |
No this is the description of the environment, where this happens ;) |
|
I think I managed to reproduce the oc_file_lock deadlock while trying to reproduce #14757 which is the oc_filecache deadlock... fun times. Steps will follow. |
|
Steps:
Expected result: the move should fail with "LockedException" Observed with mariadb mariadb-10.0.22-2.1.x86_64. The strange thing is that even though cadaver A finished already, cadaver B is still very slow to time out. I'd expect the second call to be excluded by the scanner lock. So far I had the exception only once. Depending on the timings, cadaver B might also return with "403 Forbidden" or 404. |
|
In mysql client on the CLI: My DB user was given all permissions with "GRANT ALL PERMISSIONS", so I'd expect ownCloud's connection to be able to set the transaction modes properly. |
|
Turns out I wasn't able to reproduce the original issue without this commit. It only happened once. But this variation of the steps where cadaver B (the move) runs first and implicitly scans followed by cadaver A produces the deadlock errors. Here are the updated steps:
For every file that is to be scanned I see the deadlock message appearing. Even weirder is that at the end, cadaver A still fails with "423 locked" and in the end the folder did not get renamed. Next step: @icewind1991's commit |
|
@icewind1991 I tested with your commit with both steps and could not make the deadlock issue appear again:
I tried those several times (before and after). @icewind1991 it looks like your commit fixes the issue ! (or at least my version of it) |
Which is wrong and which is what I said last week already.
That should then cause an immediate return with "locked" |
|
Please verify and then close the issue |
|
As advised, I now ran the commands in CLI mysql and after restarting the CLI session it seems that the values were kept (contrary to my previous belief). After following the steps from #20555 (comment) the "ls" call still blocks and the log still shows |
|
Retested both cases with @icewind1991's PR #24205 and with the proper transaction mode, it still works fine in a non-blocking manner (returns 423 Locked directly) |
|
It looks like this deadlock was also reproducible with two concurrent "ls" aka PROPFIND on the to-be-rescanned folder: #24260 (comment). Two concurrent PROPFINDs is more likely to be the cause of this error happening often, since multiple clients might query for changes. This, more often than a MOVE + PROPFIND. The PR above fixes that too. |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
We have a ton of following entries, they appear dozens to hundreds of times per second:
@cmonteroluque @karlitschek @PVince81 FYI - this is what is killing S3 right now. I'd vote for priority on this once.
cc @icewind1991 THX!
cc @danimo
The text was updated successfully, but these errors were encountered: