Samba external storage not working for some files

[dali99]

Steps to reproduce

1. Large image set
2. Go into folder with no thumbnail
3. Try Download the file

Expected behaviour

The file should open, and be able to download

Actual behaviour

A page with Service Unavailable shows up on Download
Error message with "Could not generate preview for this file"

Server configuration

Operating system:
Ubuntu (Docker)

Web server:
Apache (Docker)

Database:
mariadb

PHP version:
7.1.18 (Docker)

Nextcloud version: (see Nextcloud admin page)
13.0.4
Updated from an older Nextcloud/ownCloud or fresh install:
Done docker pull for a one minor version
Where did you install Nextcloud from:
Docker image https://github.com/nextcloud/docker
Signing status:

Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- twofactor_rcdevsopenotp
	- FILE_MISSING
		- l10n/.tx/config

Raw output
==========
Array
(
    [twofactor_rcdevsopenotp] => Array
        (
            [FILE_MISSING] => Array
                (
                    [l10n/.tx/config] => Array
                        (
                            [expected] => 5ad14bb93e25fd93ebbd96e1e995367db065d11953f0c3ac707962098956f1a9f5534a7dcceb834ebfb4a6b3edd7f6a8c6cbddd01c052bbb61fe951c779f2fca
                            [current] => 
                        )

                )

        )

)

List of activated apps:

App list

Enabled:
  - activity: 2.6.1
  - admin_notifications: 1.0.1
  - bruteforcesettings: 1.0.3
  - calendar: 1.6.1
  - camerarawpreviews: 0.5.3
  - checksum: 0.4.0
  - circles: 0.14.0
  - comments: 1.3.0
  - contacts: 2.1.5
  - data_request: 1.0.1
  - dav: 1.4.7
  - federatedfilesharing: 1.3.1
  - federation: 1.3.0
  - files: 1.8.0
  - files_automatedtagging: 1.3.0
  - files_downloadactivity: 1.2.0
  - files_external: 1.4.1
  - files_markdown: 2.0.4
  - files_pdfviewer: 1.2.1
  - files_rightclick: 0.8.4
  - files_sharing: 1.5.0
  - files_texteditor: 2.5.1
  - files_trashbin: 1.3.0
  - files_versions: 1.6.0
  - files_videoplayer: 1.2.0
  - firstrunwizard: 2.2.1
  - gallery: 18.0.0
  - groupfolders: 1.2.0
  - impersonate: 1.0.4
  - limit_login_to_ip: 1.0.4
  - logreader: 2.0.0
  - lookup_server_connector: 1.1.0
  - metadata: 0.6.0
  - nextcloud_announcements: 1.2.0
  - notifications: 2.1.2
  - oauth2: 1.1.1
  - password_policy: 1.3.0
  - provisioning_api: 1.3.0
  - quota_warning: 1.2.0
  - richdocuments: 2.0.9
  - serverinfo: 1.3.0
  - sharebymail: 1.3.0
  - socialsharing_diaspora: 1.0.2
  - socialsharing_email: 1.0.3
  - survey_client: 1.1.0
  - systemtags: 1.3.0
  - theming: 1.4.5
  - twofactor_backupcodes: 1.2.3
  - twofactor_totp: 1.4.1
  - twofactor_u2f: 1.5.5
  - updatenotification: 1.3.0
  - workflowengine: 1.3.0
Disabled:
  - admin_audit
  - announcementcenter
  - audioplayer
  - dashboard
  - encryption
  - end_to_end_encryption
  - files_external_gdrive
  - mindmaps
  - notes
  - twofactor_rcdevsopenotp
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.dodsorf.as"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/cloud.dodsorf.as",
        "dbtype": "mysql",
        "version": "13.0.4.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "updater.release.channel": "stable",
        "loglevel": 2,
        "maintenance": false
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
SMB

Are you using encryption: yes/no
No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser:
Tried Android Nextcloud app, and Firefox
Operating system:
Android, and Archlinux

Logs

Web server error log

Web server error log

[20/Jun/2018:14:42:28 +0000] "GET /remote.php/webdav/**REDACTED**/2015.12.20_132.JPG?downloadStartSecret=**REDACTED** HTTP/1.1" 503 9375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"98vuXe5cEUiJiND0drlw","level":4,"time":"2018-06-20T14:21:35+00:00","remoteAddr":"95.34.179.31","user":"Dandellion","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/**REDACTED**\/2015.12.20_132.JPG?downloadStartSecret=**REDACTED**","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/html\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(88): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/var\\\/www\\\/html\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#3 \\\/var\\\/www\\\/html\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:GET', Array)\\n#4 \\\/var\\\/www\\\/html\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#5 \\\/var\\\/www\\\/html\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(80): Sabre\\\\DAV\\\\Server->exec()\\n#6 \\\/var\\\/www\\\/html\\\/remote.php(164): require_once('\\\/var\\\/www\\\/html\\\/a...')\\n#7 {main}\",\"File\":\"\\\/var\\\/www\\\/html\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":337}","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:60.0) Gecko\/20100101 Firefox\/60.0","version":"13.0.4.0"}

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[nextcloud-bot]

GitMate.io thinks possibly related issues are #4730 (Storage), #3789 (External storage), #1020 (Search not working in shared (external/local storage) folders), #7898 (SMB/CIFS external storages - writing issue), and #5079 (Frequent File Locking with External Storage).

[MorrisJobke]

cc @icewind1991 @rullzer

[dali99]

UPDATE: I tried removing and readding, but the same specific files still fail.

The files work using my file manager normally.

If I try to copy the file from the external storage to just the root nextcloud folder. The file shows up as 0KB. But it has a size in the external storage folder.

Mounting through SFTP the files work

[soldier9945]

I can confirm I have the same problem, I migrated from OC 13.0.X (not sure anymore) from Ubuntu Server 14.x to a Debian Server 9 reusing both the existing data folder and database (upgraded on first use of the new NC installation).

I have tried everything and I can't find out why this won't work for the majority of files but some. Can't preview the images nor download them from the web interface.

I have SMB shares on Windows Server 2016 with Data Deduplication activated (tried to disable it but no change). Some files show Size On Disk 0KB using file explorer, those are the files that won't work in NC...

@dali99 Could you check if it's the same with your files that won't work?

[soldier9945]

I just found out that those files with 0KB are pointers and NC can't get to the files that are really stored in the V:\System Volume Information\Dedup\ Folder.

I'm trying to run a dedup job that's supposed to undo the Deduplication (here at full speed to test it, maybe stop after "-Type Unoptimization"-part if your server is in production):
PowerShell: Start-DedupJob -Volume V: -Type Unoptimization -Memory 100 -Cores 100 -Priority High -Full

UPDATE: it works.... somehow the smbclient or php checks the file size on disk and decides not to open an empty file... this is strange. I just tried opening the share with smbclient and the size of each files on a depuped file is correct and not 0kb.

[dali99]

That seems like a different issue, my files have sizes and are just raw files being served by samba on top of ZFS. And I dont think samba dedupes by default.

[soldier9945]

OK, maybe it's a different setup, but just to be clear, my file sizes are OK in NC.
In Windows Explorer I can see the same file size
BUT the "file size on disk" is 0kb for the files that are deduped.

Maybe I should start a new issue if you are certain that your setup doesn't dedupe.
UPDATE: New issue #10228

[ChristophWurst]

As the version of the software you've reported this for has reached end of life, I will close this ticket. If this is still happening after an upgrade to the latest version, feel free to reopen.