Race condition with locks in database - unique constraint violation on oc_file_locks #9305
Comments
|
bump |
|
Current Nextcloud version is 13.0.2 – does your issue still occur with NC13? |
Yes, it does. |
|
Yes, I had to move to mariadb. |
|
But I also have PostgreSQL and cannot reproduce it. 😉 |
|
@benedikt Do you use LDAP? Do you have Nextcloud behind a web proxy? Is the Nextcloud under some load (few users using it concurrently)? This is apparently some race condition, so hard to reproduce in testing environment. |
|
@benediktg it's extremely hard to reliably reproduce, however I had it happen with 2 tabs + the android client running in the same time. Using redis fixes it so it is definitely a race issue with *SQL. |
Ah, I also have redis (for transactional file locking).
no; php-fpm behind nginx here; usually I use it alone. |
MorrisJobke
changed the title
|
In more recent versions (13.0.4 and 12.0.9) we reduced drastically the amount of lock statements which should avoid most of the problems in here. Maybe also this one here is related: #8239 |
|
Crossposting from #8239: Is it possible to get a file locking debug log? There is a config switch that - once set - will log all locking attempts. Maybe we can pinpoint then the actual problem. Would one of you so kind to provide this log as we are unable to reproduce the issue on our side. You maybe want to combine this Thanks in advance. |
|
@MorrisJobke the problem persists in 13.0.4 |
This looks like it could be fixed by #10116. That one avoids, that more locks are freed than there are (the lock is usually the count of existing locks and |
|
Could we get a bug fix out soon? |
Plan is to backport it, but testing from your side may help to know if this is a fix for your problem or if we still need to check for further places as well. |
|
Just had a proper look at pull request #10116 and that wont help me, I dont use memcache for locking. |
|
Okay. Then we should check for the same in the DB provider. |
|
Just in case it helps with debug Nextcloud configuration: Config report |
|
Does the Protocoll shows references to Adapter.php LINE 115 like: If so - maybe you want to check my solution in #6160 for testing - i think the problem is in general using update and select together on one table which should work, but sometimes doesn't (at least for me). |
|
Fix is in #12366 |
|
So until I upgrade to Nextcloud 15 which has this fix, what are the consequences of this error on version 14? Will the desktop client on Windows eventually be able to upload those files? Or is this dangerous, because some files do not get synced? |
We try to backport fixes for this as well to stable14. |
* fixes #9305 by not being prone to the race condition in insertIfNotExists * fixes #6899 by not using a query that can result in a deadlock * replaces the insertIfNotExists call with an insert which is wrapped into a try-catch block * followup to #12371 Signed-off-by: Morris Jobke <hey@morrisjobke.de>
See #12413 for the fix to this issue (it fixes the race condition with the unique constraint violation and the deadlock) |
* fixes #9305 by not being prone to the race condition in insertIfNotExists * fixes #6899 by not using a query that can result in a deadlock * replaces the insertIfNotExists call with an insert which is wrapped into a try-catch block * followup to #12371 Signed-off-by: Morris Jobke <hey@morrisjobke.de>
* fixes #9305 by not being prone to the race condition in insertIfNotExists * fixes #6899 by not using a query that can result in a deadlock * replaces the insertIfNotExists call with an insert which is wrapped into a try-catch block * followup to #12371 Signed-off-by: Morris Jobke <hey@morrisjobke.de>
|
I am getting this error under 15.0.7 when I share a link, or disable the sharing of the link. I am using linux, postgresql 10.8 and apache 2.4.39. I did not get this in 13.0.5 but I got it in 14.0.12. I am using gentoo. How would I use redis to get around this? |
|
I can reproduce this every time, so its not a race condition, or at least not that I could figure out. |
|
@covici Mind to open a new ticket about this and link to this one here? |
|
I had the same problem in production today, I just restarted redis container. |
|
Hit this problem during upgrade docker container from 23-24.... now stuck on update screen |
Steps to reproduce
However, the problem does not always occur! It seems that it depends on actual load or something.
Expected behaviour
The details sidebar should slide out and user should be able to share the folder.
Actual behaviour
The details sidebar slides out, but freeze with spinners spinning forever. Nextcloud logs exception:
When user reloads the page and try it again, then it suddenly works.
Please examine Nextcloud, web server and browser logs below.
Server configuration
Operating system: Linux
Web server: nginx
Database: PostgreSQL 9.6
PHP version: 7.1.16
Nextcloud version: 12.0.6
Updated from an older Nextcloud/ownCloud or fresh install: updates
Where did you install Nextcloud from: Alpine Linux package
List of activated apps:
App list
Nextcloud configuration:
Config report
{ "system": { "datadirectory": "\/var\/lib\/nextcloud\/data", "logfile": "\/var\/log\/nextcloud\/nextcloud.log", "apps_paths": [ { "path": "\/usr\/share\/webapps\/nextcloud\/apps", "url": "\/apps", "writable": false }, { "path": "\/var\/lib\/nextcloud\/apps", "url": "\/apps-appstore", "writable": true } ], "updatechecker": false, "check_for_working_htaccess": false, "appstoreenabled": false, "memcache.local": "\\OC\\Memcache\\APCu", "instanceid": "******", "passwordsalt": "******", "secret": "******", "trusted_domains": [ "******" ], "overwrite.cli.url": "https:\/\/******", "dbtype": "pgsql", "version": "12.0.6.1", "dbname": "nextcloud", "dbhost": "postgresql", "dbport": "", "dbtableprefix": "oc_", "dbuser": "******", "dbpassword": "******", "installed": true, "ldapIgnoreNamingRules": false, "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_smtpsecure": "tls", "mail_from_address": "no-reply", "mail_domain": "******", "mail_smtpauth": 1, "mail_smtphost": "******", "mail_smtpport": "587", "mail_smtpname": "******", "mail_smtppassword": "******", "skeletondirectory": "", "theme": "", "loglevel": 2, "maintenance": false } }Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: LDAP
LDAP configuration (delete this part if not used)
LDAP config
Client configuration
Browser: various (Firefox, Safari, Chrome, …)
Operating system: various (Linux, macOS, Windows)
Logs
Web server error log
Web server error log
The first try to open Details sidebar (unsuccessful):
The second try to open Details sidebar (successful):
Nextcloud log (data/nextcloud.log)
Nextcloud log
{ "reqId":"YOsVCMaJtv7lPX2MX9Uz", "level":3, "time":"2018-04-18T14:50:28+00:00", "remoteAddr":"********", "user":"*******", "app":"no app in context", "method":"GET", "url":"\/ocs\/v2.php\/apps\/files_sharing\/api\/v1\/shares?format=json&path=%2FDocuments&shared_with_me=true", "message":"Exception: {\"Exception\":\"Doctrine\\\\DBAL\\\\Exception\\\\UniqueConstraintViolationException\",\"Message\":\"An exception occurred while executing 'INSERT INTO \\\"oc_file_locks\\\" (\\\"key\\\",\\\"lock\\\",\\\"ttl\\\") SELECT ?,?,? FROM \\\"oc_file_locks\\\" WHERE \\\"key\\\" = ? HAVING COUNT(*) = 0' with params [\\\"files\\\\\\\/737d52477f1fb583a5bfe5eb33e820da\\\", 1, 1524066628, \\\"files\\\\\\\/737d52477f1fb583a5bfe5eb33e820da\\\"]:\\n\\nSQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint \\\"lock_key_index\\\"\\nDETAIL: Key (key)=(files\\\/737d52477f1fb583a5bfe5eb33e820da) already exists.\",\"Code\":0,\"Trace\":\"#0 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/DBALException.php(128): Doctrine\\\\DBAL\\\\Driver\\\\AbstractPostgreSQLDriver->convertException('An exception oc...', Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOException))\\n#1 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/Connection.php(1015): Doctrine\\\\DBAL\\\\DBALException::driverExceptionDuringQuery(Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOPgSql\\\\Driver), Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOException), 'INSERT INTO \\\"oc...', Array)\\n#2 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Connection.php(213): Doctrine\\\\DBAL\\\\Connection->executeUpdate('INSERT INTO \\\"oc...', Array, Array)\\n#3 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Adapter.php(114): OC\\\\DB\\\\Connection->executeUpdate('INSERT INTO \\\"oc...', Array)\\n#4 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Connection.php(251): OC\\\\DB\\\\Adapter->insertIfNotExist('*PREFIX*file_lo...', Array, Array)\\n#5 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Lock\\\/DBLockingProvider.php(118): OC\\\\DB\\\\Connection->insertIfNotExist('*PREFIX*file_lo...', Array, Array)\\n#6 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Lock\\\/DBLockingProvider.php(163): OC\\\\Lock\\\\DBLockingProvider->initLockField('files\\\/737d52477...', 1)\\n#7 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Common.php(704): OC\\\\Lock\\\\DBLockingProvider->acquireLock('files\\\/737d52477...', 1)\\n#8 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1931): OC\\\\Files\\\\Storage\\\\Common->acquireLock('files\\\/Documents', 1, Object(OC\\\\Lock\\\\DBLockingProvider))\\n#9 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(2041): OC\\\\Files\\\\View->lockPath('\\\/*******\\\/files\\\/...', 1, false)\\n#10 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Node\\\/Node.php(360): OC\\\\Files\\\\View->lockFile('\\\/*******\\\/files\\\/...', 1)\\n#11 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/apps\\\/files_sharing\\\/lib\\\/Controller\\\/ShareAPIController.php(928): OC\\\\Files\\\\Node\\\\Node->lock(1)\\n#12 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/apps\\\/files_sharing\\\/lib\\\/Controller\\\/ShareAPIController.php(589): OCA\\\\Files_Sharing\\\\Controller\\\\ShareAPIController->lock(Object(OC\\\\Files\\\\Node\\\\Folder))\\n#13 [internal function]: OCA\\\\Files_Sharing\\\\Controller\\\\ShareAPIController->getShares('true', 'false', 'false', Object(OC\\\\Files\\\\Node\\\\Folder), 'false')\\n#14 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(160): call_user_func_array(Array, Array)\\n#15 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(90): OC\\\\AppFramework\\\\Http\\\\Dispatcher->executeController(Object(OCA\\\\Files_Sharing\\\\Controller\\\\ShareAPIController), 'getShares')\\n#16 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/App.php(114): OC\\\\AppFramework\\\\Http\\\\Dispatcher->dispatch(Object(OCA\\\\Files_Sharing\\\\Controller\\\\ShareAPIController), 'getShares')\\n#17 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Routing\\\/RouteActionHandler.php(47): OC\\\\AppFramework\\\\App::main('OCA\\\\\\\\Files_Shari...', 'getShares', Object(OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer), Array)\\n#18 [internal function]: OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler->__invoke(Array)\\n#19 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/lib\\\/private\\\/Route\\\/Router.php(299): call_user_func(Object(OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler), Array)\\n#20 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/ocs\\\/v1.php(78): OC\\\\Route\\\\Router->match('\\\/ocsapp\\\/apps\\\/fi...')\\n#21 \\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/ocs\\\/v2.php(23): require_once('\\\/usr\\\/share\\\/weba...')\\n#22 {main}\",\"File\":\"\\\/usr\\\/share\\\/webapps\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/Driver\\\/AbstractPostgreSQLDriver.php\",\"Line\":63}", "userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/65.0.3325.181 Safari\/537.36", "version":"12.0.6.1" }Browser log
Browser log
The first XHR request sent after click on share button:
The second XHR request sent after click on share button:
After reload, the first request for
reshares=truelooks the same.The second XHR request sent after reload and click on share button again:
The text was updated successfully, but these errors were encountered: