Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thumbnails not showing/Photos becoming unresponsive on folder with many files #643

Open
goddib opened this issue Jan 28, 2021 · 15 comments
Open
Labels
0. Needs triage Pending approval or rejection. This issue is pending approval. bug Something isn't working performances Performances issues and optimisations

Comments

@goddib
Copy link

goddib commented Jan 28, 2021

Describe the bug
I have a picture folder with about 50k photos/about 108GB and when I open it in Nextcloud it does not show any thumbnails or react to search/filter. Instead, the system becomes completely unresponsive and I cannot even navigate away while the loading animation shows. Running Nextcloud 20.0.8

To Reproduce
Steps to reproduce the behavior:

  1. Have 50k .jpg files in a folder with about 108GB total file size
  2. Open exactly that folder in "Photos" app (through "My Folders")

Expected behavior
Photos opens with thumbnails of my photos, I am able to filter/search through them by using the search functionality (.i.e. reduce number of shown pictures)

Desktop (please complete the following information):

  • OS: Windows 10
  • Browser Vivaldi/Chrome

Browser log

``` cxWeb3.js:18 MEWCX Web3 provider injected index.js:46 No OC found t.getLanguage @ index.js:46 value @ gettext.js:45 (anonymous) @ l10n.js:3 (anonymous) @ main.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ main.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ main.js:1 (anonymous) @ main.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ main.js?v=dc0e9b2b-5:1 jquery-migrate.min.js:2 JQMIGRATE: Migrate is installed, version 1.4.1 globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 o @ jquery.js:9789 0 @ jquery.js:26 n @ bootstrap:19 671 @ files_client.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ files_client.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 o @ jquery.js:9792 0 @ jquery.js:26 n @ bootstrap:19 671 @ files_client.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ files_client.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 mounted @ App.vue:117 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3945 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn @ vue.runtime.esm.js:4468 e @ vue.runtime.esm.js:4073 Ln.$mount @ vue.runtime.esm.js:8415 e._init @ vue.runtime.esm.js:5018 Ln @ vue.runtime.esm.js:5085 (anonymous) @ main.js:37 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 mounted @ App.vue:121 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3945 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn @ vue.runtime.esm.js:4468 e @ vue.runtime.esm.js:4073 Ln.$mount @ vue.runtime.esm.js:8415 e._init @ vue.runtime.esm.js:5018 Ln @ vue.runtime.esm.js:5085 (anonymous) @ main.js:37 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 mounted @ App.vue:121 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3945 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn @ vue.runtime.esm.js:4468 e @ vue.runtime.esm.js:4073 Ln.$mount @ vue.runtime.esm.js:8415 e._init @ vue.runtime.esm.js:5018 Ln @ vue.runtime.esm.js:5085 (anonymous) @ main.js:37 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:1 globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 s @ jquery.js:9789 (anonymous) @ jquery.js:26 n @ bootstrap:19 (anonymous) @ sidebar.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ sidebar.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 s @ jquery.js:9792 (anonymous) @ jquery.js:26 n @ bootstrap:19 (anonymous) @ sidebar.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ sidebar.js?v=dc0e9b2b-5:1 globals.js:60 Handlebars is deprecated: please ship your own, this will be removed in Nextcloud 20 ie @ globals.js:60 get @ globals.js:91 (anonymous) @ templates.js:2 n @ bootstrap:19 (anonymous) @ comments.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ comments.js?v=dc0e9b2b-5:1 commentstabview.js:21 Uncaught TypeError: Cannot read property 'extend' of undefined at commentstabview.js:21 at Module. (commentstabview.js:16) at n (bootstrap:19) at bootstrap:83 at comments.js?v=dc0e9b2b-5:1 (anonymous) @ commentstabview.js:21 (anonymous) @ commentstabview.js:16 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ comments.js?v=dc0e9b2b-5:1 globals.js:60 Handlebars is deprecated: please ship your own, this will be removed in Nextcloud 20 ie @ globals.js:60 get @ globals.js:91 t.default @ no-conflict.js:8 (anonymous) @ handlebars.runtime.js:60 n @ bootstrap:19 (anonymous) @ runtime.js:3 n @ bootstrap:19 (anonymous) @ item.handlebars:1 n @ bootstrap:19 (anonymous) @ files_versions.js?v=dc0e9b2b-5:1 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ files_versions.js?v=dc0e9b2b-5:1 versionstabview.js:18 Uncaught TypeError: Cannot read property 'extend' of undefined at Module. (versionstabview.js:18) at n (bootstrap:19) at bootstrap:83 at files_versions.js?v=dc0e9b2b-5:1 (anonymous) @ versionstabview.js:18 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ files_versions.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 mounted @ Notification.vue:209 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3948 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn.run @ vue.runtime.esm.js:4554 cn @ vue.runtime.esm.js:4310 (anonymous) @ vue.runtime.esm.js:1980 Ve @ vue.runtime.esm.js:1906 Promise.then (async) Qe @ vue.runtime.esm.js:1933 tt @ vue.runtime.esm.js:1990 (anonymous) @ vue.runtime.esm.js:4402 pn.update @ vue.runtime.esm.js:4544 ce.notify @ vue.runtime.esm.js:730 set @ vue.runtime.esm.js:1055 mn.set @ vue.runtime.esm.js:4631 (anonymous) @ App.vue:234 u @ runtime.js:63 (anonymous) @ runtime.js:293 (anonymous) @ runtime.js:118 O @ notifications-main.js?v=dc0e9b2b-5:303 i @ notifications-main.js?v=dc0e9b2b-5:303 Promise.then (async) O @ notifications-main.js?v=dc0e9b2b-5:303 i @ notifications-main.js?v=dc0e9b2b-5:303 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:303 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:303 _fetch @ App.vue:224 mounted @ App.vue:126 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3945 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn @ vue.runtime.esm.js:4468 e @ vue.runtime.esm.js:4073 Ln.$mount @ vue.runtime.esm.js:8415 e._init @ vue.runtime.esm.js:5018 Ln @ vue.runtime.esm.js:5085 (anonymous) @ main.js:37 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:1 globals.js:60 $ is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 mounted @ Notification.vue:209 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3948 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn.run @ vue.runtime.esm.js:4554 cn @ vue.runtime.esm.js:4310 (anonymous) @ vue.runtime.esm.js:1980 Ve @ vue.runtime.esm.js:1906 Promise.then (async) Qe @ vue.runtime.esm.js:1933 tt @ vue.runtime.esm.js:1990 (anonymous) @ vue.runtime.esm.js:4402 pn.update @ vue.runtime.esm.js:4544 ce.notify @ vue.runtime.esm.js:730 set @ vue.runtime.esm.js:1055 mn.set @ vue.runtime.esm.js:4631 (anonymous) @ App.vue:234 u @ runtime.js:63 (anonymous) @ runtime.js:293 (anonymous) @ runtime.js:118 O @ notifications-main.js?v=dc0e9b2b-5:303 i @ notifications-main.js?v=dc0e9b2b-5:303 Promise.then (async) O @ notifications-main.js?v=dc0e9b2b-5:303 i @ notifications-main.js?v=dc0e9b2b-5:303 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:303 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:303 _fetch @ App.vue:224 mounted @ App.vue:126 Ue @ vue.runtime.esm.js:1854 Kt @ vue.runtime.esm.js:4219 insert @ vue.runtime.esm.js:3139 w @ vue.runtime.esm.js:6346 (anonymous) @ vue.runtime.esm.js:6565 e._update @ vue.runtime.esm.js:3945 r @ vue.runtime.esm.js:4066 pn.get @ vue.runtime.esm.js:4479 pn @ vue.runtime.esm.js:4468 e @ vue.runtime.esm.js:4073 Ln.$mount @ vue.runtime.esm.js:8415 e._init @ vue.runtime.esm.js:5018 Ln @ vue.runtime.esm.js:5085 (anonymous) @ main.js:37 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ notifications-main.js?v=dc0e9b2b-5:1 capabilities.js:32 OC.getCapabilities is deprecated and will be removed in Nextcloud 21. See @nextcloud/capabilities getCapabilities @ capabilities.js:32 (anonymous) @ types.js:24 n @ bootstrap:19 (anonymous) @ viewer.js?v=dc0e9b2b-5:23 n @ bootstrap:19 (anonymous) @ viewer.js?v=dc0e9b2b-5:324 n @ bootstrap:19 (anonymous) @ bootstrap:83 (anonymous) @ viewer.js?v=dc0e9b2b-5:1 globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 (anonymous) @ script.js?v=dc0e9b2b-5:492 globals.js:60 jQuery is deprecated: The global jQuery is deprecated. It will be updated to v3.x in Nextcloud 21. In later versions of Nextcloud it might be removed completely. Please ship your own. ie @ globals.js:60 get @ globals.js:91 (anonymous) @ files.js?v=dc0e9b2b-5:122 Viewer.vue:217 Please do NOT wait for the DOMContentLoaded before registering your viewer handler handlers @ Viewer.vue:217 fr.run @ vue.runtime.esm.js:4568 cr @ vue.runtime.esm.js:4310 (anonymous) @ vue.runtime.esm.js:1980 Xe @ vue.runtime.esm.js:1906 Promise.then (async) Ye @ vue.runtime.esm.js:1933 tt @ vue.runtime.esm.js:1990 (anonymous) @ vue.runtime.esm.js:4402 fr.update @ vue.runtime.esm.js:4544 le.notify @ vue.runtime.esm.js:730 (anonymous) @ vue.runtime.esm.js:882 value @ Viewer.js:78 (anonymous) @ viewer.js:16 Viewer.vue:511 The following mime is already registered text/csv {__ob__: De} (anonymous) @ Viewer.vue:511 registerHandler @ Viewer.vue:508 handlers @ Viewer.vue:220 fr.run @ vue.runtime.esm.js:4568 cr @ vue.runtime.esm.js:4310 (anonymous) @ vue.runtime.esm.js:1980 Xe @ vue.runtime.esm.js:1906 Promise.then (async) Ye @ vue.runtime.esm.js:1933 tt @ vue.runtime.esm.js:1990 (anonymous) @ vue.runtime.esm.js:4402 fr.update @ vue.runtime.esm.js:4544 le.notify @ vue.runtime.esm.js:730 (anonymous) @ vue.runtime.esm.js:882 value @ Viewer.js:78 (anonymous) @ viewer.js:16 session-heartbeat.js:97 session heartbeat polling started Viewer.vue:217 Please do NOT wait for the DOMContentLoaded before registering your viewer handler handlers @ Viewer.vue:217 fr.run @ vue.runtime.esm.js:4568 cr @ vue.runtime.esm.js:4310 (anonymous) @ vue.runtime.esm.js:1980 Xe @ vue.runtime.esm.js:1906 Promise.then (async) Ye @ vue.runtime.esm.js:1933 tt @ vue.runtime.esm.js:1990 (anonymous) @ vue.runtime.esm.js:4402 fr.update @ vue.runtime.esm.js:4544 le.notify @ vue.runtime.esm.js:730 (anonymous) @ vue.runtime.esm.js:882 value @ Viewer.js:78 (anonymous) @ main.js:26 DevTools failed to load SourceMap: Could not load content for chrome-extension://mmhlniccooihdimnnjhamobppdhaolme/lib/tp/browser-polyfill.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME ```

Additional context
This issue has been haunting me ever since I started using nextcloud. will loading large folders like that ever be supported?

Thanks and cheers!

@goddib goddib added 0. Needs triage Pending approval or rejection. This issue is pending approval. bug Something isn't working labels Jan 28, 2021
@hul51u
Copy link

hul51u commented Feb 24, 2021

Sometimes my nextcloud instance has this bug, especially with "new" pictures (nextcloud server hasn't created thumbnails yet).
But my server is a nanopi-m4 (4GB ram, Dual Cortex-A72 + Quad Cortex-A53, usb3 ssd, ubuntu server), so I can't complain about that behaviour...

What are your server specs?

@yosephsa
Copy link

yosephsa commented Mar 1, 2021

@goddib Might want to add what version of nextcloud you are running.
Also do you have preview provider enabled? like ffmpeg?

@skjnldsv
Copy link
Member

skjnldsv commented Mar 2, 2021

Yeah, we added the virtual scroller optimization for Nextcloud 20 :)

  • Have 50k .jpg files in a folder with about 108GB total file size

Nonetheless, that many images is indeed crazy.
Can you make sure you cleared your cache @goddib ?

Hi! Before we continue, please do the following steps:

  1. clear your browser's cache
  2. run an occ maintenance:repair on your server

Thanks 😉

@goddib
Copy link
Author

goddib commented Mar 2, 2021

Hi all,

thank you for your feedback. First off, my server is a V-Server with 6 cores @2,2 Ghz, 16 GB Ram and 400GB storage. I feel this should be plenty for a single digit user instane of Nextcloud, right?

As of now I am running Nextcloud 20.0.8 on PHP 7.4.15 with all apps up-to-date and a green check-mark in the system overview.

I do not seem to have ffmpeg enabled - that is a PHP module, correct? I understood that is only needed for videos?

I also ran an occ maintenance:repair but without any difference. I can open the "Photos" app without any problem, I get nice previews (although loading slowly) but when I navigate to said folder, Nextcloud becomes unresponsive to a point where I need to close the browser tab (does not react to "reload").

Just as an addition: I am able to open any other folder (with way less pictures) in the app "Photos" without any problem but some delay. Maybe the slow rendering is also what crashes the system in case of the big folder?

I am thankful for any further pointers.

Thanks and cheers
goddib

@FolkertRA
Copy link

I'd like to second this and say that I am also experiencing this. Tried it last on Nextcloud 20.
It runs on a webserver from an external web hosting provider. I don't know the exact specs but Nextcloud system settings says:

CPU: | Intel Core Processor (Broadwell, IBRS) (16 cores)
Memory: | 31.26 GB

In my case, I am trying to access ~10.000 photos and some videos, totalling 70gb (most of the size comes from the videos).
For a reatively small folder (400 photos, 1gb) it works.

Some suggestions on how this could be resolved (from an end-user perspective that is):

  • Maybe it would be good to have some option to have the generation of thumbnails run on the background, with some sort of CPU throttling enabled or max only one photo at a time (single core loaded), otherwise my host will become very unhappy and probably shut down the process. I would not mind manually running occ the first time to generate the thumbnails.
  • Viewing the folder with photos should also work if the thumbnails are not yet (all) generated and not be dependent on all the thumbnails being available (or e.g. not for folders containing more than XXX photos).

P.S. Thanks for all the development effort, I realize a community-developed solution like this is limited in its resources, I hope this issue can be picked up :).

@feanor12
Copy link

I have the same problem with nextcloud 25.0.0.18. Opening the photos app and scrolling to images with low resolution thumbnails makes the server uneesponsive and I get a bad gateway error.

@Luticus
Copy link

Luticus commented Oct 26, 2022

Same here, this app on nc25 is absolutely murdering my CPU. Just opening the app sends it into overdrive. REALLY wish I could easily limit where this thing looks for photos and other media. I think that would help tremendously. I think it's checking mounted volumes, movie archives and everything. The app is called photos, I just want it to display photos albums, i don't need it to scan my entire instance and import 100,000k files per account.

@Luticus
Copy link

Luticus commented Oct 26, 2022

To put it into perspective, I disabled the photos app entirely and my instance came back to life. Like 100% speed boost across all other apps.

@feanor12
Copy link

Setting 'enable_previews' => false made my server useable again, but it is quite a drastic step.

@rarealphacat
Copy link

rarealphacat commented Mar 2, 2023

Having 600k images here and the app never worked for me, It just never loads. NC Photos on the other hand is working fine but it has to be on the phone.

@ChildLearningClub
Copy link

ChildLearningClub commented May 19, 2023

Is there some reason that media can not be index into the database like Memories does? I'm not really clear on why either that app has not been integrated better or the features from it have not been carried over to Photos? Is it possibly the sidebar and album creation that slows down Photos? Visually I can not tell what it is doing differently then Memories to slow to a crawl what happens in Memories almost instantly.

Edit: I see that when loading favorites the bottom of the screen shows a spinning loading icon and on the server only a single thread is being used. Is there some reason for only a single thread to be used?

Edit 2: There is also something with Photos that causes me to have multiple of the same favorite photo showing in some cases quadruple, I'm sure contributing to why it's so slow. Memories is not doing this.

@rosbeef
Copy link

rosbeef commented Jun 22, 2023

Is it possible to fix the preview request to some specific sizes ? (not max size)

I think the problem is that the preview automatic creation depend on the browser windows size. If the web page would ask for limited number of thumbnail, they could be generated only one time for each photos, like peviewgenerator do.

I did a script which get the size of the picture asked by the http request and use preview generator to generate thumbnail.
But the http request from the web app depend of the browser windows size so i have an infinite number of thumbnail.

@paour
Copy link

paour commented Jul 7, 2023

I have a related issue, where scrolling through photos causes NC to try to generate too many thumbnails at once, leading to an out of memory error that crashes both Apache and MariaDB.

It would make sense to queue thumbnail generation to prevent too many from running at once.

@joshtrichards
Copy link
Member

Folks on this thread:

@joshtrichards joshtrichards added the performances Performances issues and optimisations label Oct 7, 2023
@paour
Copy link

paour commented Oct 7, 2023

Using the preview_concurrency_* settings did lower the RAM usage on my server enough that this is no longer a problem, thanks for the advice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending approval or rejection. This issue is pending approval. bug Something isn't working performances Performances issues and optimisations
Projects
None yet
Development

No branches or pull requests