-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Preview for some photos not generated #36463
Comments
This issue has been present in previous nextcloud versions, too, but I didn't take time to investigate it. |
I'm getting errors like Using |
Interesting aspect. It may point to directory permissions? But this is not the cause of my problem. I have deleted the preview folder and re-generated all. |
I’d say it’s not permissions, as everything is owned by the same user, but rather that file_put_contents is trying to create a file in a folder that doesn’t exist. I patched my local installation to add a mkdir($dir, 0777, true) near the erroring line and it has fixed most of the issues.
|
Could you please point me to where you have inserted the additional |
|
Thank you very much. This change sounds reasonable and I have tried it. Very strange: And, as opposed to your issue, I still face the different paths: The shorter (old-style) folders don't exist. And why does this happen only for a relatively small subset of my photos? Any idea is welcome... :-( |
This might be linked to the issue #35319 as the photo from the user profiles are also not generated anymore |
nextcloud 26 has the same issue, but now my patch doesn't work anymore. It looks like it's blowing in a different place now:
|
I got it working on 26.0.1 with the following patch (only tested with the official Docker image): --- lib/private/Files/Node/Folder.php
+++ lib/private/Files/Node/Folder.php
@@ -162,6 +167,7 @@
$fullPath = $this->getFullPath($path);
$nonExisting = new NonExistingFolder($this->root, $this->view, $fullPath);
$this->sendHooks(['preWrite', 'preCreate'], [$nonExisting]);
+ mkdir('/var/www/html/data' . $fullPath, 0777, true);
if (!$this->view->mkdir($fullPath)) {
throw new NotPermittedException('Could not create folder');
} And if you want to build a Docker image with the patch included: FROM docker.io/nextcloud:26.0.1
WORKDIR /usr/src/nextcloud/
COPY folder.patch /folder.patch
RUN patch -p0 < /folder.patch
RUN rm /folder.patch
WORKDIR /var/www/html/ |
My version is nextcloud:26.0.3.The above repairs are not working for me |
After adding mkdir('/data' . $fullPath, 0777, true); // /data It's my local directory , I can see that the folder has been generated, but I still report this error {
"reqId": "2FMJKhb4Ct75XgSTTXhq",
"level": 3,
"time": "2023-06-24T02:47:39+00:00",
"remoteAddr": "111.224.122.208",
"user": "lieqi",
"app": "index",
"method": "GET",
"url": "/apps/photos/api/v1/preview/23471?etag=dc6a7ff4b7afbcbc0c3722444a00bc87&x=64&y=64",
"message": "Could not create folder /appdata_ock9tueu8x6i/preview/c/5/c/7/f/9/3/23471",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.51",
"version": "26.0.3.2",
"exception": {
"Exception": "OCP\\Files\\NotPermittedException",
"Message": "Could not create folder /appdata_ock9tueu8x6i/preview/c/5/c/7/f/9/3/23471",
"Code": 0,
"Trace": [
{
"file": "/config/www/nextcloud/lib/private/Files/AppData/AppData.php",
"line": 147,
"function": "newFolder",
"class": "OC\\Files\\Node\\Folder",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/Preview/Storage/Root.php",
"line": 74,
"function": "newFolder",
"class": "OC\\Files\\AppData\\AppData",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
"line": 685,
"function": "newFolder",
"class": "OC\\Preview\\Storage\\Root",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
"line": 139,
"function": "getPreviewFolder",
"class": "OC\\Preview\\Generator",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/Preview/Generator.php",
"line": 116,
"function": "generatePreviews",
"class": "OC\\Preview\\Generator",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/PreviewManager.php",
"line": 192,
"function": "getPreview",
"class": "OC\\Preview\\Generator",
"type": "->"
},
{
"file": "/config/www/nextcloud/apps/photos/lib/Controller/PreviewController.php",
"line": 162,
"function": "getPreview",
"class": "OC\\PreviewManager",
"type": "->"
},
{
"file": "/config/www/nextcloud/apps/photos/lib/Controller/PreviewController.php",
"line": 128,
"function": "fetchPreview",
"class": "OCA\\Photos\\Controller\\PreviewController",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
"line": 230,
"function": "index",
"class": "OCA\\Photos\\Controller\\PreviewController",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
"line": 137,
"function": "executeController",
"class": "OC\\AppFramework\\Http\\Dispatcher",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/AppFramework/App.php",
"line": 183,
"function": "dispatch",
"class": "OC\\AppFramework\\Http\\Dispatcher",
"type": "->"
},
{
"file": "/config/www/nextcloud/lib/private/Route/Router.php",
"line": 315,
"function": "main",
"class": "OC\\AppFramework\\App",
"type": "::"
},
{
"file": "/config/www/nextcloud/lib/base.php",
"line": 1060,
"function": "match",
"class": "OC\\Route\\Router",
"type": "->"
},
{
"file": "/config/www/nextcloud/index.php",
"line": 36,
"function": "handleRequest",
"class": "OC",
"type": "::"
}
],
"File": "/config/www/nextcloud/lib/private/Files/Node/Folder.php",
"Line": 166,
"CustomMessage": "--"
}
} |
Also encountering this issue with 200GB+ worth of photos and videos. A lot of photos and videos have broken previews, but when you click on it, the photo itself is there. |
Could this be related to the query over here? I don't see any other place in the codebase where preview folders are being deleted |
{
"reqId": "*****",
"level": 3,
"time": "2023-10-17T17:50:58+00:00",
"remoteAddr": "***.***.***.***",
"user": "*****",
"app": "PHP",
"method": "MOVE",
"url": "/remote.php/dav/uploads/*****/*****/*****",
"message": "rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#139",
"version": "25.0.8.2",
"exception": {
"Exception": "Error",
"Message": "rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#139",
"Code": 0,
"Trace": [
{
"function": "onError",
"class": "OC\\Log\\ErrorHandler",
"type": "::",
"args": [
2,
"rmdir(/var/www/data/appdata_*****/preview/1): Directory not empty",
"/var/www/html/lib/private/Files/Storage/Local.php",
139
]
},
{
"file": "/var/www/html/lib/private/Files/Storage/Local.php",
"line": 139,
"function": "rmdir",
"args": [
"/var/www/data/appdata_*****/preview/1"
]
},
{
"function": "rmdir",
"class": "OC\\Files\\Storage\\Local",
"type": "->",
"args": [
"appdata_*****/preview"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 193,
"function": "call_user_func",
"args": [
[
[
"OC\\Files\\Storage\\LocalRootStorage"
],
"rmdir"
],
"appdata_*****/preview"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 125,
"function": "doDelete",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"appdata_*****/preview",
"rmdir"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 1181,
"function": "rmdir",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"appdata_*****/preview"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 349,
"function": "basicOperation",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"rmdir",
"/appdata_*****/preview",
[
"delete"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/Folder.php",
"line": 363,
"function": "rmdir",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"/appdata_*****/preview"
]
},
{
"file": "/var/www/html/lib/private/Files/SimpleFS/SimpleFolder.php",
"line": 68,
"function": "delete",
"class": "OC\\Files\\Node\\Folder",
"type": "->",
"args": []
},
{
"file": "/var/www/html/lib/private/Preview/Watcher.php",
"line": 65,
"function": "delete",
"class": "OC\\Files\\SimpleFS\\SimpleFolder",
"type": "->",
"args": []
},
{
"file": "/var/www/html/lib/private/Preview/Watcher.php",
"line": 54,
"function": "deleteNode",
"class": "OC\\Preview\\Watcher",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/lib/private/Preview/WatcherConnector.php",
"line": 63,
"function": "postWrite",
"class": "OC\\Preview\\Watcher",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"function": "OC\\Preview\\{closure}",
"class": "OC\\Preview\\WatcherConnector",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/lib/private/Hooks/EmitterTrait.php",
"line": 106,
"function": "call_user_func_array",
"args": [
[
"Closure"
],
[
"*** sensitive parameters replaced ***"
]
]
},
{
"file": "/var/www/html/lib/private/Hooks/PublicEmitter.php",
"line": 40,
"function": "emit",
"class": "OC\\Hooks\\BasicEmitter",
"type": "->",
"args": [
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/Root.php",
"line": 143,
"function": "emit",
"class": "OC\\Hooks\\PublicEmitter",
"type": "->",
"args": [
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
},
{
"function": "emit",
"class": "OC\\Files\\Node\\Root",
"type": "->",
"args": [
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/LazyFolder.php",
"line": 72,
"function": "call_user_func_array",
"args": [
[
[
"OC\\Files\\Node\\Root"
],
"emit"
],
[
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/LazyFolder.php",
"line": 100,
"function": "__call",
"class": "OC\\Files\\Node\\LazyFolder",
"type": "->",
"args": [
"emit",
[
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/HookConnector.php",
"line": 117,
"function": "emit",
"class": "OC\\Files\\Node\\LazyFolder",
"type": "->",
"args": [
"\\OC\\Files",
"postWrite",
[
"*** sensitive parameters replaced ***"
]
]
},
{
"file": "/var/www/html/lib/private/legacy/OC_Hook.php",
"line": 106,
"function": "postWrite",
"class": "OC\\Files\\Node\\HookConnector",
"type": "->",
"args": [
[
"/****/*****/*****/*****"
]
]
},
{
"file": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
"line": 473,
"function": "emit",
"class": "OC_Hook",
"type": "::",
"args": [
"OC_Filesystem",
"post_write",
[
"/****/*****/*****/*****"
]
]
},
{
"file": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
"line": 398,
"function": "emitPostHooks",
"class": "OCA\\DAV\\Connector\\Sabre\\File",
"type": "->",
"args": [
true
]
},
{
"file": "/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php",
"line": 151,
"function": "put",
"class": "OCA\\DAV\\Connector\\Sabre\\File",
"type": "->",
"args": [
null
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
"line": 307,
"function": "createFile",
"class": "OCA\\DAV\\Connector\\Sabre\\Directory",
"type": "->",
"args": [
"*****",
null
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
"line": 133,
"function": "copyNode",
"class": "Sabre\\DAV\\Tree",
"type": "->",
"args": [
[
"OCA\\DAV\\Upload\\FutureFile"
],
[
"OCA\\DAV\\Connector\\Sabre\\Directory"
],
"*****"
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php",
"line": 163,
"function": "copy",
"class": "Sabre\\DAV\\Tree",
"type": "->",
"args": [
"uploads/*****/*****/*****",
"files/*****/****/*****/*****/*****"
]
},
{
"file": "/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php",
"line": 94,
"function": "move",
"class": "Sabre\\DAV\\Tree",
"type": "->",
"args": [
"uploads/*****/*****/*****",
"files/*****/****/*****/*****/*****"
]
},
{
"file": "/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php",
"line": 76,
"function": "performMove",
"class": "OCA\\DAV\\Upload\\ChunkingPlugin",
"type": "->",
"args": [
"uploads/*****/*****/*****",
"files/*****/****/*****/*****/*****"
]
},
{
"file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
"line": 89,
"function": "beforeMove",
"class": "OCA\\DAV\\Upload\\ChunkingPlugin",
"type": "->",
"args": [
"uploads/*****/*****/*****",
"files/*****/****/*****/*****/*****"
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
"line": 603,
"function": "emit",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
"beforeMove",
[
"uploads/*****/*****/*****",
"files/*****/****/*****/*****/*****"
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
"line": 89,
"function": "httpMove",
"class": "Sabre\\DAV\\CorePlugin",
"type": "->",
"args": [
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 472,
"function": "emit",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
"method:MOVE",
[
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 253,
"function": "invokeMethod",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 321,
"function": "start",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/dav/lib/Server.php",
"line": 360,
"function": "exec",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/dav/appinfo/v2/remote.php",
"line": 35,
"function": "exec",
"class": "OCA\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/remote.php",
"line": 172,
"args": [
"/var/www/html/apps/dav/appinfo/v2/remote.php"
],
"function": "require_once"
}
],
"File": "/var/www/html/lib/private/Log/ErrorHandler.php",
"Line": 92,
"CustomMessage": "--"
}
}
|
No hits, unfortunately. But most of my problematic photos are from 2019. If the error occured near the time of preview creation, the logs are long gone. |
Bug description
For some of my photos, no preview in no size will be generated.
I have ~60k photos in my Nextcloud file store. For most photos, the previews are generated and they work well in every app.
However, there is a bunch of photos where no previews are generated and they will not be displayed by content in any app -- of course they are shown as files without previews.
The metadata of the photos is shown on the details tab, so it's not a general file access issue.
After some investigation through the logs I think I have discovered that for these photos the preview folder is not generated correctly. The log message points to folders of this structure:
/.../nextcloud/data/appdata_xxxx/preview/284054/1536-2048-max.jpg
while it should be of this structure:
/.../nextcloud/data/appdata_xxxx/preview/5/6/4/0/1/4/1/0/284054/1536-2048-max.jpg
According to my investigations, the responsible code is found in
server/lib/private/Preview/Storage/Root.php
, functionpublic function getFolder(string $name): ISimpleFolder
.Why does this code fail for some files while it doesn't for all the others?!?
What can I do to resolve this issue?!?
Steps to reproduce
occ preview:generate-all
(the photo itself of course is not broken, it is clean when I open it in any program on Windows or Linux -- I tried many)
Expected behavior
Both doesn't happen.
Installation method
Community Manual installation with Archive
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.1
Web server
Apache (supported)
Database engine version
MySQL
Is this bug present after an update or on a fresh install?
None
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
List of activated Apps
# sudo -u www-data php /mnt/data/var/www/nextcloud/occ app:list Enabled: - activity: 2.17.0 - admin_audit: 1.15.0 - audioplayer: 3.3.1 - bookmarks: 12.0.0 - bruteforcesettings: 2.5.0 - calendar: 4.2.2 - camerarawpreviews: 0.8.0 - checksum: 1.2.0 - circles: 25.0.0 - cloud_federation_api: 1.8.0 - cloud_py_api: 0.1.4 - comments: 1.15.0 - contacts: 5.0.3 - contactsinteraction: 1.6.0 - dashboard: 7.5.0 - dav: 1.24.0 - deck: 1.8.3 - event_update_notification: 2.0.0 - external: 5.0.0 - extract: 1.3.5 - federatedfilesharing: 1.15.0 - federation: 1.15.0 - files: 1.20.1 - files_downloadactivity: 1.15.0 - files_external: 1.17.0 - files_fulltextsearch: 25.0.0 - files_fulltextsearch_tesseract: 25.0.0 - files_markdown: 2.3.6 - files_mindmap: 0.0.27 - files_pdfviewer: 2.6.0 - files_rightclick: 1.4.0 - files_sharing: 1.17.0 - files_trashbin: 1.15.0 - files_versions: 1.18.0 - firstrunwizard: 2.14.0 - flow_notifications: 1.5.0 - fulltextsearch: 25.0.0 - fulltextsearch_elasticsearch: 25.0.0 - gpxmotion: 0.1.0 - gpxpod: 5.0.4 - groupfolders: 13.1.0 - logreader: 2.10.0 - lookup_server_connector: 1.13.0 - mail: 2.2.2 - maps: 0.2.4 - mediadc: 0.3.3 - memories: 4.10.3 - metadata: 0.17.0 - music: 1.8.1 - news: 20.0.1 - nextcloud_announcements: 1.14.0 - notes: 4.6.0 - notifications: 2.13.1 - oauth2: 1.13.0 - password_policy: 1.15.0 - passwords: 2023.1.23 - photos: 2.0.1 - previewgenerator: 5.1.1 - privacy: 1.9.0 - provisioning_api: 1.15.0 - quickaccesssorting: 1.1.4 - recommendations: 1.4.0 - related_resources: 1.0.4 - richdocuments: 7.1.0 - serverinfo: 1.15.0 - settings: 1.7.0 - sharebymail: 1.15.0 - side_menu: 3.5.2 - spreed: 15.0.3 - support: 1.8.0 - survey_client: 1.13.0 - suspicious_login: 4.3.0 - systemtags: 1.15.0 - talked: 0.4.0 - tasks: 0.14.5 - text: 3.6.0 - theming: 2.0.1 - twofactor_backupcodes: 1.14.0 - updatenotification: 1.15.0 - user_status: 1.5.0 - video_converter: 1.0.5 - viewer: 1.9.0 - weather_status: 1.5.0 - welcome: 1.0.6 - workflow_ocr: 1.25.3 - workflowengine: 2.7.0 Disabled: - audioplayer_editor: 0.3.0 - audioplayer_sonos: 1.3.0 - bookmarks_fulltextsearch: 1.2.0 - duplicatefinder: 0.0.15 - encryption - epubreader: 1.4.7 - facerecognition: 0.9.5 - files_antivirus: 4.0.2 - ransomware_protection: 1.14.0 - recognize: 3.3.6 - twofactor_totp - user_ldap: 1.10.2
Nextcloud Signing status
Nextcloud Logs
The same message formatted:
[PHP] Fehler: Error: file_put_contents(/.../nextcloud/data/appdata_xxxxx/preview/284054/1536-2048-max.jpg): Failed to open stream: No such file or directory at /.../nextcloud/lib/private/Files/Storage/Local.php#311 at <>
OC\Log\ErrorHandler::onError()
file_put_contents()
OC\Files\Storage\Local->file_put_contents()
OC\Files\Storage\Wrapper\Wrapper->file_put_contents()
OC\Files\View->basicOperation()
OC\Files\View->file_put_contents()
OC\Files\Node\Folder->newFile()
OC\Files\SimpleFS\NewSimpleFile->putContent()
OC\Preview\Generator->getMaxPreview()
OC\Preview\Generator->generatePreviews()
OC\Preview\Generator->getPreview()
OC\PreviewManager->getPreview()
OC\Core\Controller\PreviewController->fetchPreview()
OC\Core\Controller\PreviewController->getPreviewByFileId()
OC\AppFramework\Http\Dispatcher->executeController()
OC\AppFramework\Http\Dispatcher->dispatch()
OC\AppFramework\App::main()
OC\Route\Router->match()
OC::handleRequest()
GET /nextcloud/core/preview?fileId=284054&c=46f47f6cce3eee1cb4d3653abfd7b9d4&x=375&y=375&forceIcon=0&a=0
from 2a02:560:56ac:2a00:e1b0:b437:9572:eb8d by thomas at 2023-01-31T19:32:33+00:00
The text was updated successfully, but these errors were encountered: