Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Source file could not be loaded #440

Open
rustyscottweber opened this issue Jan 18, 2024 · 16 comments
Open

Error: Source file could not be loaded #440

rustyscottweber opened this issue Jan 18, 2024 · 16 comments

Comments

@rustyscottweber
Copy link

rustyscottweber commented Jan 18, 2024

While running nohup /root/occ preview:generate-all &

2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Documents
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Dungeons&Dragons
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Camera
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/DCIM
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Documents
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Download
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Facebook
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Lumii
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Messenger
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/PhotoScan
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/PhotosEditor
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Pictures
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Raw
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Screenshots
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/ToonMe
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Voice
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Voila
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Templates
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia/Important Recordings
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia/Lake Powell 2023
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Documents
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Photos
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Templates
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia/Important Recordings
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia/Lake Powell 2023
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Documents
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Photos
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Templates
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents/GCW Family Documents
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS
Error: source file could not be loaded
Error: source file could not be loaded
Error: source file could not be loaded

@rustyscottweber
Copy link
Author

Looks like this also occurs on

nohup ./occ preview:pre-generate &
[1] 7301
nohup: ignoring input and appending output to 'nohup.out'
root@cloud:~# tail -f ./nohup.out 
/usr/share/nginx/nextcloud ~
Error: source file could not be loaded

@rustyscottweber
Copy link
Author

Looking at the verbose output of either command shows a more interesting tail:

2024-01-19T01:10:39+00:00 Generating previews for /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS/021003.msg
Error: source file could not be loaded
2024-01-19T01:14:30+00:00 Generating previews for /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS/030206.msg
Error: source file could not be loaded

So, while attempting to process .msg files, which are plain text emails, the preview generator takes about 2-5 minutes before returning an error that the source file could not be loaded. At 2-5 minutes a file, when there thousands of them, the scan will essentially never finish running.

@rustyscottweber
Copy link
Author

rustyscottweber commented Jan 23, 2024

It would appear that, if left to it's own designs, this bug will cause enough preview generation processes to pile up that it will eventually crash the server.

root@cloud:~# ps aux | grep -i preview:generate-all
www-data    3518  0.0  0.0   2892   912 ?        Ss   Jan19   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data    3520  0.0  0.3 483780 39552 ?        S    Jan19   3:45 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
root       15503  0.0  0.0   7372  2356 ?        S    Jan19   0:00 bash /root/occ preview:generate-all -vvv
root       15504  0.0  0.0  11668  4576 ?        S    Jan19   0:00 sudo -u www-data php occ preview:generate-all -vvv
root       15505  0.0  0.0  11668   104 pts/1    Ss+  Jan19   0:00 sudo -u www-data php occ preview:generate-all -vvv
www-data   15506  0.0  1.1 479772 144292 pts/1   S    Jan19   3:54 php occ preview:generate-all -vvv
www-data   25687  0.0  0.0   2892  1000 ?        Ss   Jan20   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   25688  0.0  1.3 478860 160828 ?       S    Jan20   4:13 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   38955  0.0  0.0   2892   980 ?        Ss   Jan21   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   38957  0.1  1.3 478992 165524 ?       S    Jan21   4:02 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   53123  0.0  0.0   2892  1000 ?        Ss   Jan22   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   53126  0.0  1.1 455924 139964 ?       S    Jan22   0:03 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   66849  0.0  0.0   2892  1004 ?        Ss   03:30   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   66852  0.0  1.1 455924 140048 ?       S    03:30   0:10 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
root       73334  0.0  0.0   6624  2404 pts/2    S+   14:06   0:00 grep --color=auto -i preview:generate-all

All of these separate running cron jobs cause PHP to time out meaning that the server cannot field requests to users.

@rustyscottweber
Copy link
Author

It turns out that part of the errors here are that I had an outdated cron job running preview:generate-all in stead of preview:pre-generate. I changed my cron job, which should stop the server from eventually crashing. However, the failure to generate certain previews still prevails making the pre-generate cron job run far longer than it need to.

@st3iny
Copy link
Member

st3iny commented Feb 6, 2024

The problem might be that the failed previews are tried to be generated again and again.

I have no experience with the preview provider for msg files and can't promise anything. You could disable it for now if you are not depending on it. I imagine having previews for email files is not critical.

@rustyscottweber
Copy link
Author

Indeed, part of the issue is that previews are attempted again and again and turning off previews for . Msg files would solve that. How does one disable specific previews?

@joshtrichards
Copy link
Member

Previews are managed through a combination of enabledPreviewProviders and, in some cases, some apps register preview providers themselves.

We don't do any .msg handling by default (at least not intentionally).

What apps do you have installed (occ app:list)?

Anything like the following?

https://github.com/nextcloud/files_emailviewer

(Though I don't see a preview provider in it).

@mellow2012
Copy link

Hi
we have the same problem with .msg files
our installed apps are:
`Enabled:

  • activity: 2.21.1
  • admin_audit: 1.19.0
  • analytics: 4.14.0
  • audioplayer: 3.4.1
  • bookmarks: 14.2.2
  • bruteforcesettings: 2.9.0
  • calendar: 4.7.14
  • calendar_resource_management: 0.8.0
  • cfg_share_links: 5.1.0
  • circles: 29.0.0-dev
  • cloud_federation_api: 1.12.0
  • comments: 1.19.0
  • contacts: 6.0.0
  • cospend: 1.6.1
  • dashboard: 7.9.0
  • dav: 1.30.1
  • drawio: 3.0.2
  • external: 5.4.0
  • federatedfilesharing: 1.19.0
  • federation: 1.19.0
  • files: 2.1.0
  • files_antivirus: 5.5.7
  • files_automatedtagging: 1.19.0
  • files_external: 1.21.0
  • files_pdfviewer: 2.10.0
  • files_reminders: 1.2.0
  • files_retention: 1.18.0
  • files_sharing: 1.21.0
  • files_trashbin: 1.19.0
  • files_versions: 1.22.0
  • firstrunwizard: 2.18.0
  • forms: 4.2.4
  • guests: 3.1.0
  • integration_giphy: 1.0.11
  • logreader: 2.14.0
  • lookup_server_connector: 1.17.0
  • mail: 3.7.6
  • memegen: 1.1.0
  • notifications: 2.17.0
  • notify_push: 0.7.0
  • oauth2: 1.17.0
  • password_policy: 1.19.0
  • passwords: 2024.7.20
  • phonetrack: 0.8.1
  • photos: 2.5.0
  • polls: 7.1.4
  • previewgenerator: 5.6.0
  • privacy: 1.13.0
  • provisioning_api: 1.19.0
  • quota_warning: 1.19.0
  • related_resources: 1.4.0
  • richdocuments: 8.4.4
  • serverinfo: 1.19.0
  • settings: 1.12.0
  • sharebymail: 1.19.0
  • sharelisting: 1.2.0
  • sharereview: 1.0.2
  • side_menu: 3.13.1
  • spreed: 19.0.7
  • systemtags: 1.19.0
  • tables: 0.7.5
  • tasks: 0.16.0
  • terms_of_service: 2.5.0
  • text: 3.10.1
  • theming: 2.4.0
  • theming_customcss: 1.17.0
  • translate: 2.2.0
  • twofactor_backupcodes: 1.18.0
  • updatenotification: 1.19.1
  • user_ldap: 1.20.0
  • user_status: 1.9.0
  • viewer: 2.3.0
  • workflow_ocr: 1.29.0
  • workflowengine: 2.11.0
    Disabled:
  • contactsinteraction: 1.10.0 (installed 1.6.0)
  • deck: 1.13.1 (installed 1.13.1)
  • encryption: 2.17.0
  • files_downloadlimit: 2.0.0 (installed 1.1.0)
  • nextcloud_announcements: 1.18.0 (installed 1.12.0)
  • recommendations: 2.1.0 (installed 1.2.0)
  • support: 1.12.0 (installed 1.6.0)
  • survey_client: 1.17.0 (installed 1.11.0)
  • suspicious_login: 7.0.0
  • twofactor_totp: 11.0.0-dev
  • weather_status: 1.9.0 (installed 1.3.0)
    `

@rustyscottweber
Copy link
Author

@mellow2012 , how are you executing the script? What command is in your cronjob?

@mellow2012
Copy link

i am executing it from the comandline on the ubuntu server like occ preview:generate-all —path=the path

but i found today in the dtabase in oc_mimetypes the
aplication/vns.ms-outlook
but i dont no from where it came

@rustyscottweber
Copy link
Author

rustyscottweber commented Aug 7, 2024

Do you get the same hang using "preview:pre-generate" multiple times? preview:pre-generate will not allow multiple runs to happen at the same time and I suspect that they now mark failed files so that they aren't re-attempted, but I can't remember. can you run pre-generate twice, in stead of generate-all and report back your finding? Generate all will force the regeneration of files that already have a preview generated. Using pre-generate doesn't regenerate previews if one already exists. So, you might get the error/timeout the first time around, but you shouldn't get it the next time around because the file should already be marked as attempted. The first run took me longer than a week to finish because I had so many files, but the second run took a minute. This means you still won't have a preview for the msg files, but at least everything else will be generated.

@mellow2012
Copy link

at the moment i need it for all

@rustyscottweber
Copy link
Author

preview:pre-generate will generate it for all, it just won't regenerate or re-attempt on previous failure.

@mellow2012
Copy link

yes i know
but i need at the moment the outher one
so i dont witch of my apps is the problem with the .msg file

@ETES-Stuttgart
Copy link

ETES-Stuttgart commented Aug 22, 2024

We have the same issue I guess, but in addition to the preview-generation, the normal cronjob throws those Error: source file could not be loaded-errors as well... I guess?

Using the occ-command for the preview-generation manually for specific users/folders, I found out, the error appears for *.msg-files as mentioned in this issue before.

[root@cloud]# sudo -u apache php /var/www/html/nextcloud/occ preview:generate-all --path="/<username>/files/some/path/" -vv
2024-08-22T13:52:44+02:00 Scanning folder /<username>/files/some/path
2024-08-22T13:54:05+02:00 Generating previews for /<username>/files/some/path/somefile.docx
2024-08-22T13:54:05+02:00 Generating previews for /<username>/files/some/path/anotherfile_1.msg
Error: source file could not be loaded
2024-08-22T13:54:06+02:00 Generating previews for /<username>/files/some/path/anotherfile_2.pdf
2024-08-22T13:54:06+02:00 Generating previews for /<username>/files/some/path/anotherfile_3.msg
Error: source file could not be loaded
2024-08-22T13:54:10+02:00 Generating previews for /<username>/files/some/path/anotherfile_4.msg
Error: source file could not be loaded

Any news about getting rid of those errors?

Also, as mentioned above, the error is also thrown, when the "normal" cronjob (sudo -u apache php /var/www/html/nextcloud/cron.php) runs; anyway, not always and also with different amounts of appearance of the error - so I am not exactly sure, if those error errors have the same source (those *msg-files)?
Anyway to verify this? Can you increase the verbosity of the cron?

@joshtrichards
Copy link
Member

I'm wondering if this is related to nextcloud/server#41803, nextcloud/server#41867, and nextcloud/server#41868.

These were added for files_emailviewer. Though in the final, msg files aren't text/plain so I wouldn't expect us to process them 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants