-
-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
php occ files:scan exceeds maximal number of parameters while deleting files with PostgreSQL #25732
Comments
Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :) |
I can verify that this issue (or at least a very similar one) is still valid in NC21.0.2 (derived from docker image
Server configurationOperating system: Docker on QNAP Web server: nginx (behind another nginx reverse proxy) Database: PostgreSQL 13.3 PHP version: 7.4.20 Nextcloud version: 21.0.2 Updated from an older Nextcloud/ownCloud or fresh install: Where did you install Nextcloud from: Signing status: Signing status
List of activated apps: App list
Nextcloud configuration: Config reportI replaced the actual domain name with `my.domain.name`.
Are you using external storage, if yes which one: local+smb/cifs Are you using encryption: no Are you using an external user-backend, if yes which one: Not anymore (SAML SSO was in use for some time before) LogsWeb server error logWeb server error log
Nextcloud log (data/nextcloud.log)Nextcloud logThe logfiles have become very large, primarily due to entries taking up multiple *megabytes* for a single entry like the following originally ~15MB line. Notice the `[SNIP]`s to keep the report size incheck.
Browser logBrowser logNo browser interaction |
Could you please add some steps how to reproduce? Thanks! |
I did not yet have time to do a full from-scratch repro, but the easiest way to trigger the exception in my running instance is:
It takes a minute or two for the problem to occur, but so far it seems to happen every single time. |
How many files do you have on your external storages and how many Nextcloud users that have access to those external storages? |
There are many files but only a few users with access |
How many is many (approx.)? |
I should have expected that question: Roughly a million files distributed over some 65k folders. Number of users is <10. |
Hm... that doesn't sound unreasonable, should be actually handleable. |
cc @icewind1991 |
This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions. |
Still a problem. I suspect a solution would be changing up server/lib/private/Files/Cache/Cache.php Lines 605 to 608 in 7a0b374
A workaround would be to recreate a few empty folders in the data directory with the names of the folders in the root of the deleted file, and then calling files:scan again. This will cause nextcloud to do one query per folder instead of all folders at the same time. |
Do you mind creating a PR with a patch for discussion? |
I'm currently not able to, but I'll make one when I'm available in about a week |
This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions. |
Could this be reopened? The issue still exists. |
@vijfhoek any update here? :) |
I have Samba storages connected to NC. There were large changes in the files in the storage from clients outside NC. So when NC reconciled everything, it had to delete a huge chunk which likely was giving this error. |
@szaimen I must admit that this issue kind of ended up on the backburner here 😅 I've just implemented the fix I suggested, currently trying to reproduce the issue again so I can confirm it indeed fixes the issue. |
I managed to reproduce the issue, and confirmed my proposed fix fixes the issue. I have also done some research on what would be a good limit, based on the maximum number of parameters in a query:
Interestingly, during some very unscientific testing I've found that a parameter count to 2048 has a significantly better performance on postgresql (2:58 min for the full scan) versus 32768 (3:07 min). This could be fully attributed to some variable (disk cache, background processes, whatever) but at least it means it isn't a major performance hit :) I'll submit a pull request in a bit! |
How to use GitHub
Steps to reproduce
Expected behaviour
The filecache is cleaned up.
Actual behaviour
I'm getting this exception:
Server configuration
Operating system:
Debian Buster
Web server:
--
Database:
PostgrSQL
PHP version:
PHP 7.3.x
Nextcloud version: (see Nextcloud admin page)
20.0.7
Updated from an older Nextcloud/ownCloud or fresh install:
Updated
Where did you install Nextcloud from:
Signing status:
No errors have been found.
``
List of activated apps:
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: local/smb/sftp/...
Are you using encryption: yes/no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP configuration (delete this part if not used)
LDAP config
Client configuration
Browser:
Operating system:
Logs
Web server error log
Web server error log
Nextcloud log (data/nextcloud.log)
Nextcloud log
Browser log
Browser log
The text was updated successfully, but these errors were encountered: