[Bug]: /tmp dir upload not working #30843

gaalos · 2022-01-25T18:22:38Z

⚠️ Before submitting, please verify the following: ⚠️

This is a bug, not a question or a configuration issue.
This issue is not already reported on Github (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
I agree to follow Nextcloud's Code of Conduct

Bug description

When user upload files, the part files ares uploaded on USERNAME/uploads/web-file-upload-xxxx insted of configured upload dir

Steps to reproduce

upload_tmp_dir = /YOUR PATH
in config.php 'tempdirectory' => '/YOUR PATH',
restart your services
give access to your webuser to /YOUR PATH
restart your services
upload big files
...

Expected behavior

Part files is uploaded inside tmp files (better storage performances)
Then file is moved to floder (slowest disk)

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 7.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

Default internal user-backend
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.XX.EU",
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/cloud.choulou.tk",
        "dbtype": "mysql",
        "version": "23.0.0.10",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "maintenance": false,
        "loglevel": 2,
        "theme": "",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mysql.utf8mb4": true,
        "app_install_overwrite": [
            "files_external_gdrive"
        ],
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": "true",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 9500,
            "timeout": 0
        },
        "tempdirectory": "\/mnt\/ramdisk\/"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - admin_audit: 1.13.0
  - circles: 23.0.0
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contacts: 4.0.7
  - contactsinteraction: 1.4.0
  - dav: 1.21.0
  - external: 3.10.2
  - federatedfilesharing: 1.13.0
  - federation: 1.13.0
  - files: 1.18.0
  - files_downloadactivity: 1.12.0
  - files_external: 1.15.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - groupfolders: 11.1.2
  - impersonate: 1.10.0
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - nextcloud_announcements: 1.12.0
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - onlyoffice: 7.3.0
  - password_policy: 1.13.0
  - passwords: 2022.1.10
  - photos: 1.5.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - recommendations: 1.2.0
  - richdocuments: 5.0.1
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - support: 1.6.0
  - survey_client: 1.11.0
  - systemtags: 1.13.0
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - updatenotification: 1.13.0
  - viewer: 1.7.0
  - workflowengine: 2.5.0
Disabled:
  - dashboard
  - encryption
  - user_ldap
  - user_status
  - weather_status

Nextcloud Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- core
	- INVALID_HASH
		- core/js/mimetypelist.js

Raw output
==========
Array
(
    [core] => Array
        (
            [INVALID_HASH] => Array
                (
                    [core/js/mimetypelist.js] => Array
                        (
                            [expected] => 94195a260a005dac543c3f6aa504f1b28e0078297fe94a4f52f012c16c109f0323eecc9f767d6949f860dfe454625fcaf1dc56f87bb8350975d8f006bbbdf14a
                            [current] => 1b07fb272efa65a10011ed52a6e51260343c5de2a256e1ae49f180173e2b6684ccf90d1af3c19fa97c31d42914866db46e3216883ec0d6a82cec0ad5529e78b1
                        )

                )

        )

)

Nextcloud Logs

no logs

Additional info

no add

solracsf · 2022-01-26T14:48:23Z

AFAIK, this folder is not used for chunked uploads.

server/apps/files/js/file-upload.js

Lines 261 to 330 in a145edd

    
           		var chunkFolderPromise; 
        
           		if ($.support.blobSlice 
        
           			&& this.uploader.fileUploadParam.maxChunkSize 
        
           			&& this.getFile().size > this.uploader.fileUploadParam.maxChunkSize 
        
           		) { 
        
           			data.isChunked = true; 
        
           			chunkFolderPromise = this.uploader.davClient.createDirectory( 
        
           				'uploads/' + OC.getCurrentUser().uid + '/' + this.getId() 
        
           			); 
        
           			// TODO: if fails, it means same id already existed, need to retry 
        
           		} else { 
        
           			chunkFolderPromise = $.Deferred().resolve().promise(); 
        
           			var mtime = this.getFile().lastModified; 
        
           			if (mtime) { 
        
           				data.headers['X-OC-Mtime'] = mtime / 1000; 
        
           			} 
        
           		} 
        
           		// wait for creation of the required directory before uploading 
        
           		return Promise.all([folderPromise, chunkFolderPromise]).then(function() { 
        
           			if (self.aborted !== true) { 
        
           				data.submit(); 
        
           			} 
        
           		}, function() { 
        
           			self.abort(); 
        
           		}); 
        
           	}, 
        
           	/** 
        
           	 * Process end of transfer 
        
           	 */ 
        
           	done: function() { 
        
           		if (!this.data.isChunked) { 
        
           			return $.Deferred().resolve().promise(); 
        
           		} 
        
           		var uid = OC.getCurrentUser().uid; 
        
           		var mtime = this.getFile().lastModified; 
        
           		var size = this.getFile().size; 
        
           		var headers = {}; 
        
           		if (mtime) { 
        
           			headers['X-OC-Mtime'] = mtime / 1000; 
        
           		} 
        
           		if (size) { 
        
           			headers['OC-Total-Length'] = size; 
        
           		} 
        
           		return this.uploader.davClient.move( 
        
           			'uploads/' + uid + '/' + this.getId() + '/.file', 
        
           			'files/' + uid + '/' + OC.joinPaths(this.getFullPath(), this.getFileName()), 
        
           			true, 
        
           			headers 
        
           		); 
        
           	}, 
        
           	_deleteChunkFolder: function() { 
        
           		// delete transfer directory for this upload 
        
           		this.uploader.davClient.remove( 
        
           			'uploads/' + OC.getCurrentUser().uid + '/' + this.getId() 
        
           		); 
        
           	}, 
        
           	_delete: function() { 
        
           		if (this.data.isChunked) { 
        
           			this._deleteChunkFolder() 
        
           		} 
        
           		this.deleteUpload(); 
        
           	},

One option may be disabling chunked uploads (set to --value 0) or by setting a big value which is, by default, set to 10MB.

server/apps/files/lib/App.php

Lines 57 to 60 in 0447b53

    
           $maxChunkSize = (int)\OC::$server->getConfig()->getAppValue('files', 'max_chunk_size', 10 * 1024 * 1024); 
        
           $appConfig['files'] = [ 
        
           	'max_chunk_size' => $maxChunkSize 
        
           ];

See https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html#adjust-chunk-size-on-nextcloud-side

In ALL cases, it will be your upload_tmp_dir, set in PHP, that will be used. Set it to your desired path.

gaalos · 2022-01-27T12:32:08Z

@acsfer thx for your reply ! Chunk is nice for uploading big files ;).
I think add tmp-chunk upload might be useful in some cases.
For exemple use ramdisk or fast storage for construct file and upload it into the use storage.
What do you mean

In ALL cases, it will be your upload_tmp_dir, set in PHP, that will be used. Set it to your desired path.

When upload files the part of chunk are stored in upload_tmp_dir and after that moved to the upload user folder and after that de file is construct ?

Torbins · 2022-02-18T13:34:53Z

tmp dir is not involved at all for chunk storage. If you want to speed up uploads, enable 'localstorage.allowsymlinks' in the config and manually symlink uploads folder to the separate drive.

gaalos · 2022-02-20T21:42:17Z

Hi thx for your reply
But with S3 storage it's not localstorage ? :/

See more: - nextcloud/server#19682 - nextcloud/server#30843

szaimen · 2023-01-23T12:59:13Z

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

pchang388 · 2023-02-12T01:09:21Z

This is still occuring for me on 25.0.3-apache version. The doc states that it should change both for php chunks and regular tmp files but it does not apply to php chunks. I want to do chunk/parts upload to a local fs and then move them to a mounted nfs for long term retention and disk capacity on the NFS. The reason for this being that large file uploads (yes I changed all relevant settings according to tuning guides) fails at ~2 GB size using NFS backend. It appears related to the slowness of certain file operations on NFS like move, copy, etc..

I am looking into setting up data dir on local and using external storage plugin instead - might solve the issue: https://docs.nextcloud.com/server/25/admin_manual/configuration_files/external_storage_configuration_gui.html

szaimen · 2023-02-12T10:28:00Z

Hi, did you check if setting these two values makes it work?
https://github.com/nextcloud/all-in-one/blob/8b8059b6d2c4230e0250c288951df44756d1069a/Containers/nextcloud/entrypoint.sh#L374-L376

gaalos added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jan 25, 2022

solracsf mentioned this issue Jan 26, 2022

S3 upload: troubles with big files (temp files) #29589

Closed

haeho7 added a commit to haeho7/docker-images that referenced this issue Dec 20, 2022

nextcloud: tempdirectory not working

e81dc23

See more: - nextcloud/server#19682 - nextcloud/server#30843

szaimen added the needs info label Jan 23, 2023

szaimen closed this as completed Mar 6, 2023

kesselb mentioned this issue Feb 18, 2024

[Bug]: files are written twice during upload #43636

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: /tmp dir upload not working #30843

[Bug]: /tmp dir upload not working #30843

gaalos commented Jan 25, 2022 •

edited

Loading

solracsf commented Jan 26, 2022 •

edited

Loading

gaalos commented Jan 27, 2022 •

edited

Loading

Torbins commented Feb 18, 2022

gaalos commented Feb 20, 2022

szaimen commented Jan 23, 2023

pchang388 commented Feb 12, 2023 •

edited

Loading

szaimen commented Feb 12, 2023

[Bug]: /tmp dir upload not working #30843

[Bug]: /tmp dir upload not working #30843

Comments

gaalos commented Jan 25, 2022 • edited Loading

⚠️ Before submitting, please verify the following: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

Are you using an external user-backend?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

solracsf commented Jan 26, 2022 • edited Loading

gaalos commented Jan 27, 2022 • edited Loading

Torbins commented Feb 18, 2022

gaalos commented Feb 20, 2022

szaimen commented Jan 23, 2023

pchang388 commented Feb 12, 2023 • edited Loading

szaimen commented Feb 12, 2023

gaalos commented Jan 25, 2022 •

edited

Loading

solracsf commented Jan 26, 2022 •

edited

Loading

gaalos commented Jan 27, 2022 •

edited

Loading

pchang388 commented Feb 12, 2023 •

edited

Loading