Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Preview for some photos not generated #36463

Closed
6 of 9 tasks
SpamReceiver opened this issue Jan 31, 2023 · 16 comments · Fixed by #41067
Closed
6 of 9 tasks

[Bug]: Preview for some photos not generated #36463

SpamReceiver opened this issue Jan 31, 2023 · 16 comments · Fixed by #41067
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 25-feedback bug feature: previews and thumbnails

Comments

@SpamReceiver
Copy link
Contributor

SpamReceiver commented Jan 31, 2023

⚠️ This issue respects the following points: ⚠️

  • This is a bug, not a question or a configuration/webserver/proxy issue.
  • This issue is not already reported on Github (I've searched it).
  • Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
  • Nextcloud Server is running on 64bit capable CPU, PHP and OS.
  • I agree to follow Nextcloud's Code of Conduct.

Bug description

For some of my photos, no preview in no size will be generated.

I have ~60k photos in my Nextcloud file store. For most photos, the previews are generated and they work well in every app.

However, there is a bunch of photos where no previews are generated and they will not be displayed by content in any app -- of course they are shown as files without previews.

The metadata of the photos is shown on the details tab, so it's not a general file access issue.

After some investigation through the logs I think I have discovered that for these photos the preview folder is not generated correctly. The log message points to folders of this structure:
/.../nextcloud/data/appdata_xxxx/preview/284054/1536-2048-max.jpg
while it should be of this structure:
/.../nextcloud/data/appdata_xxxx/preview/5/6/4/0/1/4/1/0/284054/1536-2048-max.jpg

According to my investigations, the responsible code is found in server/lib/private/Preview/Storage/Root.php, function public function getFolder(string $name): ISimpleFolder.

Why does this code fail for some files while it doesn't for all the others?!?
What can I do to resolve this issue?!?

Steps to reproduce

  1. If only I knew...
  2. Use my photo collection
  3. occ preview:generate-all
  4. Click on "broken" photo in "files" app
    (the photo itself of course is not broken, it is clean when I open it in any program on Windows or Linux -- I tried many)

Expected behavior

  • in the "files" app, a small preview icon is shown
  • after clicking on the photo, a large preview is shown

Both doesn't happen.

  • a placeholder icon is shown
  • "The image cannot be loaded" is shown

Installation method

Community Manual installation with Archive

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

# sudo -u www-data php /mnt/data/var/www/nextcloud/occ config:list system
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "x.y.z",
            "x",
            "192.168.1.1"
        ],
        "overwrite.cli.url": "https:\/\/x.y.z\/nextcloud",
        "htaccess.RewriteBase": "\/nextcloud",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.3.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "filelocking.enabled": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "loglevel": 0,
        "app_install_overwrite": [
            "bookmarks_fulltextsearch",
            "quickaccesssorting",
            "ocr",
            "files_downloadactivity",
            "checksum",
            "audioplayer_editor",
            "epubreader",
            "gpxmotion",
            "previewgenerator",
            "audioplayer_sonos",
            "camerarawpreviews",
            "apporder",
            "duplicatefinder",
            "files_markdown",
            "talked",
            "video_converter",
            "extract",
            "files_ebookreader",
            "talk_simple_poll"
        ],
        "loglevel:9": "",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "preview_max_memory": 4096,
        "preview_max_filesize_image": 256,
        "jpeg_quality": "60",
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\Movie",
            "OC\\Preview\\MKV",
            "OC\\Preview\\MP4",
            "OC\\Preview\\AVI"
        ],
        "theme": "",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "default_phone_region": "DE",
        "trashbin_retention_obligation": "90, auto",
        "memories.exiftool": "\/mnt\/data\/var\/www\/nextcloud\/apps\/memories\/exiftool-bin\/exiftool-amd64-glibc",
        "memories.ffmpeg_path": "\/usr\/bin\/ffmpeg",
        "memories.ffprobe_path": "\/usr\/bin\/ffprobe",
        "memories.transcoder": "\/mnt\/data\/var\/www\/nextcloud\/apps\/memories\/exiftool-bin\/go-vod-amd64",
        "memories.no_transcode": false,
        "memories.qsv": true
    }
}

List of activated Apps

# sudo -u www-data php /mnt/data/var/www/nextcloud/occ app:list
Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - audioplayer: 3.3.1
  - bookmarks: 12.0.0
  - bruteforcesettings: 2.5.0
  - calendar: 4.2.2
  - camerarawpreviews: 0.8.0
  - checksum: 1.2.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - cloud_py_api: 0.1.4
  - comments: 1.15.0
  - contacts: 5.0.3
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - deck: 1.8.3
  - event_update_notification: 2.0.0
  - external: 5.0.0
  - extract: 1.3.5
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_downloadactivity: 1.15.0
  - files_external: 1.17.0
  - files_fulltextsearch: 25.0.0
  - files_fulltextsearch_tesseract: 25.0.0
  - files_markdown: 2.3.6
  - files_mindmap: 0.0.27
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - flow_notifications: 1.5.0
  - fulltextsearch: 25.0.0
  - fulltextsearch_elasticsearch: 25.0.0
  - gpxmotion: 0.1.0
  - gpxpod: 5.0.4
  - groupfolders: 13.1.0
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - mail: 2.2.2
  - maps: 0.2.4
  - mediadc: 0.3.3
  - memories: 4.10.3
  - metadata: 0.17.0
  - music: 1.8.1
  - news: 20.0.1
  - nextcloud_announcements: 1.14.0
  - notes: 4.6.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - passwords: 2023.1.23
  - photos: 2.0.1
  - previewgenerator: 5.1.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - quickaccesssorting: 1.1.4
  - recommendations: 1.4.0
  - related_resources: 1.0.4
  - richdocuments: 7.1.0
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - side_menu: 3.5.2
  - spreed: 15.0.3
  - support: 1.8.0
  - survey_client: 1.13.0
  - suspicious_login: 4.3.0
  - systemtags: 1.15.0
  - talked: 0.4.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - updatenotification: 1.15.0
  - user_status: 1.5.0
  - video_converter: 1.0.5
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - welcome: 1.0.6
  - workflow_ocr: 1.25.3
  - workflowengine: 2.7.0
Disabled:
  - audioplayer_editor: 0.3.0
  - audioplayer_sonos: 1.3.0
  - bookmarks_fulltextsearch: 1.2.0
  - duplicatefinder: 0.0.15
  - encryption
  - epubreader: 1.4.7
  - facerecognition: 0.9.5
  - files_antivirus: 4.0.2
  - ransomware_protection: 1.14.0
  - recognize: 3.3.6
  - twofactor_totp
  - user_ldap: 1.10.2

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"WZAyV8M8jHYZHTrLtOVG","level":3,"time":"2023-01-31T19:23:00+00:00","remoteAddr":"2a02:560:56ac:2a00:e1b0:b437:9572:eb8d","user":"aaa","app":"PHP","method":"GET","url":"/nextcloud/core/preview?fileId=284057&c=2f7c0b38ddf242dcddb5fd0e6d13be55&x=375&y=375&forceIcon=0&a=0","message":"file_put_contents(/.../nextcloud/data/appdata_xxxxx/preview/284057/1536-2048-max.jpg): Failed to open stream: No such file or directory at /.../nextcloud/lib/private/Files/Storage/Local.php#311","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","version":"25.0.3.2","exception":{"Exception":"Error","Message":"file_put_contents(/.../nextcloud/data/appdata_xxxxx/preview/284057/1536-2048-max.jpg): Failed to open stream: No such file or directory at /.../nextcloud/lib/private/Files/Storage/Local.php#311","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/.../nextcloud/lib/private/Files/Storage/Local.php","line":311,"function":"file_put_contents"},{"file":"/.../nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":258,"function":"file_put_contents","class":"OC\\Files\\Storage\\Local","type":"->"},{"file":"/.../nextcloud/lib/private/Files/View.php","line":1179,"function":"file_put_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/.../nextcloud/lib/private/Files/View.php","line":705,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/.../nextcloud/lib/private/Files/Node/Folder.php","line":192,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/.../nextcloud/lib/private/Files/SimpleFS/NewSimpleFile.php","line":121,"function":"newFile","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/.../nextcloud/lib/private/Preview/Generator.php","line":363,"function":"putContent","class":"OC\\Files\\SimpleFS\\NewSimpleFile","type":"->"},{"file":"/.../nextcloud/lib/private/Preview/Generator.php","line":162,"function":"getMaxPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/.../nextcloud/lib/private/Preview/Generator.php","line":114,"function":"generatePreviews","class":"OC\\Preview\\Generator","type":"->"},{"file":"/.../nextcloud/lib/private/PreviewManager.php","line":185,"function":"getPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/.../nextcloud/core/Controller/PreviewController.php","line":144,"function":"getPreview","class":"OC\\PreviewManager","type":"->"},{"file":"/.../nextcloud/core/Controller/PreviewController.php","line":113,"function":"fetchPreview","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/.../nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"getPreviewByFileId","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/.../nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/.../nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/.../nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/.../nextcloud/lib/base.php","line":1047,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/.../nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/.../nextcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"id":"63d96a950405a"}

The same message formatted:

[PHP] Fehler: Error: file_put_contents(/.../nextcloud/data/appdata_xxxxx/preview/284054/1536-2048-max.jpg): Failed to open stream: No such file or directory at /.../nextcloud/lib/private/Files/Storage/Local.php#311 at <>

  1. <>
    OC\Log\ErrorHandler::onError()
  2. /.../nextcloud/lib/private/Files/Storage/Local.php line 311
    file_put_contents()
  3. /.../nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 258
    OC\Files\Storage\Local->file_put_contents()
  4. /.../nextcloud/lib/private/Files/View.php line 1179
    OC\Files\Storage\Wrapper\Wrapper->file_put_contents()
  5. /..../nextcloud/lib/private/Files/View.php line 705
    OC\Files\View->basicOperation()
  6. /..../nextcloud/lib/private/Files/Node/Folder.php line 192
    OC\Files\View->file_put_contents()
  7. /.../nextcloud/lib/private/Files/SimpleFS/NewSimpleFile.php line 121
    OC\Files\Node\Folder->newFile()
  8. /.../nextcloud/lib/private/Preview/Generator.php line 363
    OC\Files\SimpleFS\NewSimpleFile->putContent()
  9. /.../nextcloud/lib/private/Preview/Generator.php line 162
    OC\Preview\Generator->getMaxPreview()
  10. /.../nextcloud/lib/private/Preview/Generator.php line 114
    OC\Preview\Generator->generatePreviews()
  11. /.../nextcloud/lib/private/PreviewManager.php line 185
    OC\Preview\Generator->getPreview()
  12. /.../nextcloud/core/Controller/PreviewController.php line 144
    OC\PreviewManager->getPreview()
  13. /.../nextcloud/core/Controller/PreviewController.php line 113
    OC\Core\Controller\PreviewController->fetchPreview()
  14. /.../nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 225
    OC\Core\Controller\PreviewController->getPreviewByFileId()
  15. /.../nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 133
    OC\AppFramework\Http\Dispatcher->executeController()
  16. /..../nextcloud/lib/private/AppFramework/App.php line 172
    OC\AppFramework\Http\Dispatcher->dispatch()
  17. /.../nextcloud/lib/private/Route/Router.php line 298
    OC\AppFramework\App::main()
  18. /.../nextcloud/lib/base.php line 1047
    OC\Route\Router->match()
  19. /.../nextcloud/index.php line 36
    OC::handleRequest()

GET /nextcloud/core/preview?fileId=284054&c=46f47f6cce3eee1cb4d3653abfd7b9d4&x=375&y=375&forceIcon=0&a=0
from 2a02:560:56ac:2a00:e1b0:b437:9572:eb8d by thomas at 2023-01-31T19:32:33+00:00


### Additional info

_No response_
@SpamReceiver SpamReceiver added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jan 31, 2023
@SpamReceiver
Copy link
Contributor Author

This issue has been present in previous nextcloud versions, too, but I didn't take time to investigate it.

@kassner
Copy link

kassner commented Feb 19, 2023

I'm getting errors like file_put_contents(/var/www/html/data/appdata_oc1gkz6atyze/preview/4/3/a/3/b/b/1/178181/946-2048-max.png): Failed to open stream: No such file or directory at /var/www/html/lib/private/Files/Storage/Local.php#311. I'm not sure why the extra subfolders when yours was just a single ID, but I've figured that if I mkdir -p the folder in question and try again, the preview gets generated.

Using docker.io/nextcloud:25.0.3 via Podman.

@SpamReceiver
Copy link
Contributor Author

Interesting aspect. It may point to directory permissions?

But this is not the cause of my problem. I have deleted the preview folder and re-generated all.
The issue remains as reported above.

@kassner
Copy link

kassner commented Feb 21, 2023 via email

@SpamReceiver
Copy link
Contributor Author

Could you please point me to where you have inserted the additional mkdir command?
I'd like to try the same patch.

@kassner
Copy link

kassner commented Feb 22, 2023

--- a/lib/private/Files/Storage/Local.php
+++ b/lib/private/Files/Storage/Local.php
@@ -309,7 +309,10 @@ public function file_put_contents($path, $data) {
 		if ($this->unlinkOnTruncate) {
 			$this->unlink($path);
 		}
-		$result = file_put_contents($this->getSourcePath($path), $data);
+		$sourcePath = $this->getSourcePath($path);
+		$pathDir = dirname($sourcePath);
+		$this->mkdir($pathDir);
+		$result = file_put_contents($sourcePath, $data);
 		umask($oldMask);
 		return $result;
 	}

@SpamReceiver
Copy link
Contributor Author

SpamReceiver commented Feb 22, 2023

Thank you very much. This change sounds reasonable and I have tried it.

Very strange:
I still face the same error message as originally reported!
As if the mkdir hasn't any effect...

And, as opposed to your issue, I still face the different paths:
/.../nextcloud/data/appdata_xxxx/preview/284054/1536-2048-max.jpg
instead the expected:
/.../nextcloud/data/appdata_xxxx/preview/5/6/4/0/1/4/1/0/284054/1536-2048-max.jpg

The shorter (old-style) folders don't exist.
Why does it try to create them, and why does it fail doing so?

And why does this happen only for a relatively small subset of my photos?

Any idea is welcome... :-(

@Daryes
Copy link

Daryes commented Feb 25, 2023

This might be linked to the issue #35319 as the photo from the user profiles are also not generated anymore

@kassner
Copy link

kassner commented May 23, 2023

nextcloud 26 has the same issue, but now my patch doesn't work anymore. It looks like it's blowing in a different place now:

{
    "reqId":"xxx",
    "level":3,
    "time":"2023-05-23T18:43:00+00:00",
    "remoteAddr":"xxx",
    "user":"xxx",
    "app":"index",
    "method":"GET",
    "url":"/apps/photos/api/v1/preview/190054?etag=xxx&x=512&y=512",
    "message":"Could not create folder",
    "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0",
    "version":"26.0.1.1",
    "exception":{
        "Exception":"OCP\\\\Files\\\\NotPermittedException",
        "Message":"Could not create folder",
        "Code":0,
        "Trace":[
            {"file":"/var/www/html/lib/private/Files/AppData/AppData.php","line":147,"function":"newFolder","class":"OC\\\\Files\\\\Node\\\\Folder","type":"->","args":["3/0/5/9/a/6/6/190054"]},
            {"file":"/var/www/html/lib/private/Preview/Storage/Root.php","line":74,"function":"newFolder","class":"OC\\\\Files\\\\AppData\\\\AppData","type":"->","args":["3/0/5/9/a/6/6/190054"]},
            {"file":"/var/www/html/lib/private/Preview/Generator.php","line":685,"function":"newFolder","class":"OC\\\\Preview\\\\Storage\\\\Root","type":"->","args":["190054"]},
            {"file":"/var/www/html/lib/private/Preview/Generator.php","line":139,"function":"getPreviewFolder","class":"OC\\\\Preview\\\\Generator","type":"->","args":[["OC\\\\Files\\\\Node\\\\File"]]},
            {"file":"/var/www/html/lib/private/Preview/Generator.php","line":116,"function":"generatePreviews","class":"OC\\\\Preview\\\\Generator","type":"->","args":[["OC\\\\Files\\\\Node\\\\File"],[[512,512,false,"fill"]],"image/jpeg"]},
            {"file":"/var/www/html/lib/private/PreviewManager.php","line":192,"function":"getPreview","class":"OC\\\\Preview\\\\Generator","type":"->","args":[["OC\\\\Files\\\\Node\\\\File"],512,512,false,"fill",null]},
            {"file":"/var/www/html/apps/photos/lib/Controller/PreviewController.php","line":162,"function":"getPreview","class":"OC\\\\PreviewManager","type":"->","args":[["OC\\\\Files\\\\Node\\\\File"],512,512]},
            {"file":"/var/www/html/apps/photos/lib/Controller/PreviewController.php","line":128,"function":"fetchPreview","class":"OCA\\\\Photos\\\\Controller\\\\PreviewController","type":"->","args":[["OC\\\\Files\\\\Node\\\\File"],512,512]},
            {"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"index","class":"OCA\\\\Photos\\\\Controller\\\\PreviewController","type":"->","args":[190054,512,512]},
            {"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->","args":[["OCA\\\\Photos\\\\Controller\\\\PreviewController"],"index"]},
            {"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->","args":[["OCA\\\\Photos\\\\Controller\\\\PreviewController"],"index"]},
            {"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\\\AppFramework\\\\App","type":"::","args":["OCA\\\\Photos\\\\Controller\\\\PreviewController","index",["OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer"],["190054","photos.preview.index"]]},
            {"file":"/var/www/html/lib/base.php","line":1056,"function":"match","class":"OC\\\\Route\\\\Router","type":"->","args":["/apps/photos/api/v1/preview/190054"]},
            {"file":"/var/www/html/index.php","line":36,"function":"handleRequest","class":"OC","type":"::","args":[]}
        ],
        "File":"/var/www/html/lib/private/Files/Node/Folder.php",
        "Line":166,
        "CustomMessage":"--"
    }
}

@kassner
Copy link

kassner commented May 23, 2023

I got it working on 26.0.1 with the following patch (only tested with the official Docker image):

--- lib/private/Files/Node/Folder.php
+++ lib/private/Files/Node/Folder.php
@@ -162,6 +167,7 @@
                        $fullPath = $this->getFullPath($path);
                        $nonExisting = new NonExistingFolder($this->root, $this->view, $fullPath);
                        $this->sendHooks(['preWrite', 'preCreate'], [$nonExisting]);
+                       mkdir('/var/www/html/data' . $fullPath, 0777, true);
                        if (!$this->view->mkdir($fullPath)) {
                                throw new NotPermittedException('Could not create folder');
                        }

And if you want to build a Docker image with the patch included:

FROM docker.io/nextcloud:26.0.1

WORKDIR /usr/src/nextcloud/
COPY folder.patch /folder.patch
RUN patch -p0 < /folder.patch
RUN rm /folder.patch
WORKDIR /var/www/html/

@Ador-able
Copy link

My version is nextcloud:26.0.3.The above repairs are not working for me

@Ador-able
Copy link

Ador-able commented Jun 24, 2023

After adding

mkdir('/data' . $fullPath, 0777, true); // /data It's my local directory

, I can see that the folder has been generated, but I still report this error

{
    "reqId": "2FMJKhb4Ct75XgSTTXhq",
    "level": 3,
    "time": "2023-06-24T02:47:39+00:00",
    "remoteAddr": "111.224.122.208",
    "user": "lieqi",
    "app": "index",
    "method": "GET",
    "url": "/apps/photos/api/v1/preview/23471?etag=dc6a7ff4b7afbcbc0c3722444a00bc87&x=64&y=64",
    "message": "Could not create folder /appdata_ock9tueu8x6i/preview/c/5/c/7/f/9/3/23471",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51",
    "version": "26.0.3.2", 
    "exception": {
        "Exception": "OCP\\Files\\NotPermittedException",
        "Message": "Could not create folder /appdata_ock9tueu8x6i/preview/c/5/c/7/f/9/3/23471",
        "Code": 0,
        "Trace": [
            {
                "file": "/config/www/nextcloud/lib/private/Files/AppData/AppData.php",
                "line": 147,
                "function": "newFolder",
                "class": "OC\\Files\\Node\\Folder",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/Preview/Storage/Root.php",
                "line": 74,
                "function": "newFolder",
                "class": "OC\\Files\\AppData\\AppData",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
                "line": 685,
                "function": "newFolder",
                "class": "OC\\Preview\\Storage\\Root",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
                "line": 139,
                "function": "getPreviewFolder",
                "class": "OC\\Preview\\Generator",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
                "line": 116,
                "function": "generatePreviews",
                "class": "OC\\Preview\\Generator",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/PreviewManager.php",
                "line": 192,
                "function": "getPreview",
                "class": "OC\\Preview\\Generator",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/apps/photos/lib/Controller/PreviewController.php",
                "line": 162,
                "function": "getPreview",
                "class": "OC\\PreviewManager",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/apps/photos/lib/Controller/PreviewController.php",
                "line": 128,
                "function": "fetchPreview",
                "class": "OCA\\Photos\\Controller\\PreviewController",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
                "line": 230,
                "function": "index",
                "class": "OCA\\Photos\\Controller\\PreviewController",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
                "line": 137,
                "function": "executeController",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/AppFramework/App.php",
                "line": 183,
                "function": "dispatch",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/lib/private/Route/Router.php",
                "line": 315,
                "function": "main",
                "class": "OC\\AppFramework\\App",
                "type": "::"
            },
            {
                "file": "/config/www/nextcloud/lib/base.php",
                "line": 1060,
                "function": "match",
                "class": "OC\\Route\\Router",
                "type": "->"
            },
            {
                "file": "/config/www/nextcloud/index.php",
                "line": 36,
                "function": "handleRequest",
                "class": "OC",
                "type": "::"
            }
        ],
        "File": "/config/www/nextcloud/lib/private/Files/Node/Folder.php",
        "Line": 166,
        "CustomMessage": "--"
    }
}

@mannyvergel
Copy link

Also encountering this issue with 200GB+ worth of photos and videos. A lot of photos and videos have broken previews, but when you click on it, the photo itself is there.

@akhil1508
Copy link
Contributor

Could this be related to the query over here?

I don't see any other place in the codebase where preview folders are being deleted

@akhil1508
Copy link
Contributor

akhil1508 commented Oct 17, 2023

  • I do see an error in lib/private/Preview/Watcher.php that is likely related. There is an attempt to delete parent folders of a preview folder of a file(it seems like a path issue) which maybe should not happen. I see it on a server running on NC 25 but I don't see any relevant changes in later versions related to this error..

  • The stack trace looks like:

{
	"reqId": "*****",
	"level": 3,
	"time": "2023-10-17T17:50:58+00:00",
	"remoteAddr": "***.***.***.***",
	"user": "*****",
	"app": "PHP",
	"method": "MOVE",
	"url": "/remote.php/dav/uploads/*****/*****/*****",
	"message": "rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#139",
	"version": "25.0.8.2",
	"exception": {
		"Exception": "Error",
		"Message": "rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#139",
		"Code": 0,
		"Trace": [
			{
				"function": "onError",
				"class": "OC\\Log\\ErrorHandler",
				"type": "::",
				"args": [
					2,
					"rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty",
					"/var/www/html/lib/private/Files/Storage/Local.php",
					139
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Storage/Local.php",
				"line": 139,
				"function": "rmdir",
				"args": [
					"/var/www/data/appdata_*****/preview/1"
				]
			},
			{
				"function": "rmdir",
				"class": "OC\\Files\\Storage\\Local",
				"type": "->",
				"args": [
					"appdata_*****/preview"
				]
			},
			{
				"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
				"line": 193,
				"function": "call_user_func",
				"args": [
					[
						[
							"OC\\Files\\Storage\\LocalRootStorage"
						],
						"rmdir"
					],
					"appdata_*****/preview"
				]
			},
			{
				"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
				"line": 125,
				"function": "doDelete",
				"class": "OCA\\Files_Trashbin\\Storage",
				"type": "->",
				"args": [
					"appdata_*****/preview",
					"rmdir"
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/View.php",
				"line": 1181,
				"function": "rmdir",
				"class": "OCA\\Files_Trashbin\\Storage",
				"type": "->",
				"args": [
					"appdata_*****/preview"
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/View.php",
				"line": 349,
				"function": "basicOperation",
				"class": "OC\\Files\\View",
				"type": "->",
				"args": [
					"rmdir",
					"/appdata_*****/preview",
					[
						"delete"
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Node/Folder.php",
				"line": 363,
				"function": "rmdir",
				"class": "OC\\Files\\View",
				"type": "->",
				"args": [
					"/appdata_*****/preview"
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/SimpleFS/SimpleFolder.php",
				"line": 68,
				"function": "delete",
				"class": "OC\\Files\\Node\\Folder",
				"type": "->",
				"args": []
			},
			{
				"file": "/var/www/html/lib/private/Preview/Watcher.php",
				"line": 65,
				"function": "delete",
				"class": "OC\\Files\\SimpleFS\\SimpleFolder",
				"type": "->",
				"args": []
			},
			{
				"file": "/var/www/html/lib/private/Preview/Watcher.php",
				"line": 54,
				"function": "deleteNode",
				"class": "OC\\Preview\\Watcher",
				"type": "->",
				"args": [
					"*** sensitive parameters replaced ***"
				]
			},
			{
				"file": "/var/www/html/lib/private/Preview/WatcherConnector.php",
				"line": 63,
				"function": "postWrite",
				"class": "OC\\Preview\\Watcher",
				"type": "->",
				"args": [
					"*** sensitive parameters replaced ***"
				]
			},
			{
				"function": "OC\\Preview\\{closure}",
				"class": "OC\\Preview\\WatcherConnector",
				"type": "->",
				"args": [
					"*** sensitive parameters replaced ***"
				]
			},
			{
				"file": "/var/www/html/lib/private/Hooks/EmitterTrait.php",
				"line": 106,
				"function": "call_user_func_array",
				"args": [
					[
						"Closure"
					],
					[
						"*** sensitive parameters replaced ***"
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Hooks/PublicEmitter.php",
				"line": 40,
				"function": "emit",
				"class": "OC\\Hooks\\BasicEmitter",
				"type": "->",
				"args": [
					"\\OC\\Files",
					"postWrite",
					[
						"*** sensitive parameters replaced ***"
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Node/Root.php",
				"line": 143,
				"function": "emit",
				"class": "OC\\Hooks\\PublicEmitter",
				"type": "->",
				"args": [
					"\\OC\\Files",
					"postWrite",
					[
						"*** sensitive parameters replaced ***"
					]
				]
			},
			{
				"function": "emit",
				"class": "OC\\Files\\Node\\Root",
				"type": "->",
				"args": [
					"\\OC\\Files",
					"postWrite",
					[
						"*** sensitive parameters replaced ***"
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Node/LazyFolder.php",
				"line": 72,
				"function": "call_user_func_array",
				"args": [
					[
						[
							"OC\\Files\\Node\\Root"
						],
						"emit"
					],
					[
						"\\OC\\Files",
						"postWrite",
						[
							"*** sensitive parameters replaced ***"
						]
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Node/LazyFolder.php",
				"line": 100,
				"function": "__call",
				"class": "OC\\Files\\Node\\LazyFolder",
				"type": "->",
				"args": [
					"emit",
					[
						"\\OC\\Files",
						"postWrite",
						[
							"*** sensitive parameters replaced ***"
						]
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/Files/Node/HookConnector.php",
				"line": 117,
				"function": "emit",
				"class": "OC\\Files\\Node\\LazyFolder",
				"type": "->",
				"args": [
					"\\OC\\Files",
					"postWrite",
					[
						"*** sensitive parameters replaced ***"
					]
				]
			},
			{
				"file": "/var/www/html/lib/private/legacy/OC_Hook.php",
				"line": 106,
				"function": "postWrite",
				"class": "OC\\Files\\Node\\HookConnector",
				"type": "->",
				"args": [
					[
						"/****/*****/*****/*****"
					]
				]
			},
			{
				"file": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
				"line": 473,
				"function": "emit",
				"class": "OC_Hook",
				"type": "::",
				"args": [
					"OC_Filesystem",
					"post_write",
					[
						"/****/*****/*****/*****"
					]
				]
			},
			{
				"file": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
				"line": 398,
				"function": "emitPostHooks",
				"class": "OCA\\DAV\\Connector\\Sabre\\File",
				"type": "->",
				"args": [
					true
				]
			},
			{
				"file": "/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php",
				"line": 151,
				"function": "put",
				"class": "OCA\\DAV\\Connector\\Sabre\\File",
				"type": "->",
				"args": [
					null
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
				"line": 307,
				"function": "createFile",
				"class": "OCA\\DAV\\Connector\\Sabre\\Directory",
				"type": "->",
				"args": [
					"*****",
					null
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
				"line": 133,
				"function": "copyNode",
				"class": "Sabre\\DAV\\Tree",
				"type": "->",
				"args": [
					[
						"OCA\\DAV\\Upload\\FutureFile"
					],
					[
						"OCA\\DAV\\Connector\\Sabre\\Directory"
					],
					"*****"
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
				"line": 163,
				"function": "copy",
				"class": "Sabre\\DAV\\Tree",
				"type": "->",
				"args": [
					"uploads/*****/*****/*****",
					"files/*****/****/*****/*****/*****"
				]
			},
			{
				"file": "/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php",
				"line": 94,
				"function": "move",
				"class": "Sabre\\DAV\\Tree",
				"type": "->",
				"args": [
					"uploads/*****/*****/*****",
					"files/*****/****/*****/*****/*****"
				]
			},
			{
				"file": "/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php",
				"line": 76,
				"function": "performMove",
				"class": "OCA\\DAV\\Upload\\ChunkingPlugin",
				"type": "->",
				"args": [
					"uploads/*****/*****/*****",
					"files/*****/****/*****/*****/*****"
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
				"line": 89,
				"function": "beforeMove",
				"class": "OCA\\DAV\\Upload\\ChunkingPlugin",
				"type": "->",
				"args": [
					"uploads/*****/*****/*****",
					"files/*****/****/*****/*****/*****"
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
				"line": 603,
				"function": "emit",
				"class": "Sabre\\DAV\\Server",
				"type": "->",
				"args": [
					"beforeMove",
					[
						"uploads/*****/*****/*****",
						"files/*****/****/*****/*****/*****"
					]
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
				"line": 89,
				"function": "httpMove",
				"class": "Sabre\\DAV\\CorePlugin",
				"type": "->",
				"args": [
					[
						"Sabre\\HTTP\\Request"
					],
					[
						"Sabre\\HTTP\\Response"
					]
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
				"line": 472,
				"function": "emit",
				"class": "Sabre\\DAV\\Server",
				"type": "->",
				"args": [
					"method:MOVE",
					[
						[
							"Sabre\\HTTP\\Request"
						],
						[
							"Sabre\\HTTP\\Response"
						]
					]
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
				"line": 253,
				"function": "invokeMethod",
				"class": "Sabre\\DAV\\Server",
				"type": "->",
				"args": [
					[
						"Sabre\\HTTP\\Request"
					],
					[
						"Sabre\\HTTP\\Response"
					]
				]
			},
			{
				"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
				"line": 321,
				"function": "start",
				"class": "Sabre\\DAV\\Server",
				"type": "->",
				"args": []
			},
			{
				"file": "/var/www/html/apps/dav/lib/Server.php",
				"line": 360,
				"function": "exec",
				"class": "Sabre\\DAV\\Server",
				"type": "->",
				"args": []
			},
			{
				"file": "/var/www/html/apps/dav/appinfo/v2/remote.php",
				"line": 35,
				"function": "exec",
				"class": "OCA\\DAV\\Server",
				"type": "->",
				"args": []
			},
			{
				"file": "/var/www/html/remote.php",
				"line": 172,
				"args": [
					"/var/www/html/apps/dav/appinfo/v2/remote.php"
				],
				"function": "require_once"
			}
		],
		"File": "/var/www/html/lib/private/Log/ErrorHandler.php",
		"Line": 92,
		"CustomMessage": "--"
	}
}
  • The number of these errors and the number of preview errors seem similar to me
  • Could it be that this method in Watcher.php actually deletes folders in the preview root folder that it shouldn't ? (when it succeeds, not when this error happens)
  • @mannyvergel @Ador-able @SpamReceiver Could you please check your logs to see if there are similar errors too for you? :)

@SpamReceiver
Copy link
Contributor Author

  • I do see an error in lib/private/Preview/Watcher.php that is likely related. There is an attempt to delete parent folders of a preview folder of a file(it seems like a path issue) which maybe should not happen. I see it on a server running on NC 25 but I don't see any relevant changes in later versions related to this error..

    • The stack trace looks like:
      [snip]

    • The number of these errors and the number of preview errors seem similar to me

    • Could it be that this method in Watcher.php actually deletes folders in the preview root folder that it shouldn't ? (when it succeeds, not when this error happens)

    • @mannyvergel @Ador-able @SpamReceiver Could you please check your logs to see if there are similar errors too for you? :)

# grep "Directory not" nextcloud.log*
# grep "Watcher.php" nextcloud.log*

No hits, unfortunately.

But most of my problematic photos are from 2019. If the error occured near the time of preview creation, the logs are long gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 25-feedback bug feature: previews and thumbnails
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants