Unable to upload 6GB file to nextcloud when SWIFT object storage is used as primary storage or as external storage

[rontrompert]

Steps to reproduce

1. Unable to upload 6GB file to nextcloud when SWIFT object storage is used as primary storage or as external storage

Expected behaviour

Tell us what should happen
File is successfully uploaded

Actual behaviour

Tell us what happens instead
Server responds with "Internal server error"

Server configuration

Operating system:
CentOS 7.3.1611

Web server:
apache 2.4.6

Database:
mariadb-5.5

PHP version:
7.0

Nextcloud version: (see Nextcloud admin page)
11

Updated from an older Nextcloud/ownCloud or fresh install:
fresh

Where did you install Nextcloud from:
tarball

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors have been found.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Enabled:

• activity: 2.4.1
• comments: 1.1.0
• dav: 1.1.1
• federatedfilesharing: 1.1.1
• federation: 1.1.1
• files: 1.6.1
• files_external: 1.1.2
• files_pdfviewer: 1.0.1
• files_sharing: 1.1.1
• files_texteditor: 2.2
• files_trashbin: 1.1.0
• files_versions: 1.4.0
• files_videoplayer: 1.0.0
• gallery: 16.0.0
• lookup_server_connector: 1.0.0
• nextcloud_announcements: 1.0
• notifications: 1.0.1
• password_policy: 1.1.0
• provisioning_api: 1.1.0
• sharebymail: 1.0.1
• systemtags: 1.1.3
• theming: 1.1.1
• twofactor_backupcodes: 1.0.0
• updatenotification: 1.1.1
• workflowengine: 1.1.1

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

{
"system": {
"instanceid": "ocwdxao1plbh",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"necen.data.surfsara.nl",
"portal.cdi.surfsara.nl"
],
"overwrite.cli.url": "https://necen.data.surfsara.nl",
"overwritehost": "necen.data.surfsara.nl",
"overwriteprotocol": "https",
"forwarded_for_headers": [
"HTTP_X_FORWARDED_FOR",
"HTTP_FORWARDED_FOR"
],
"memcache.local": "\OC\Memcache\APCu",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "145.100.5.230",
"port": 6379,
"timeout": 0,
"password": "REMOVED SENSITIVE VALUE",
"dbindex": 0
},
"filelocking.enabled": "true",
"trusted_proxies": [
"145.100.5.230"
],
"mail_domain": "necen.data.surfsara.nl",
"version": "11.0.3.2",
"appcodechecker": false,
"updatechecker": false,
"installed": true,
"enable_previews": true,
"preview_max_x": 1024,
"preview_max_y": 1024,
"preview_max_filesize_image": 100,
"enabledPreviewProviders": [
"OC\Preview\PNG",
"OC\Preview\JPEG",
"OC\Preview\GIF",
"OC\Preview\BMP",
"OC\Preview\XBitmap",
"OC\Preview\TXT",
"OC\Preview\MarkDown"
],
"appstoreenabled": false,
"has_internet_connection": false,
"auth.bruteforce.protection.enabled": true,
"enable_avatars": true,
"dbtype": "mysql",
"dbname": "necen_nc",
"dbhost": "145.100.5.230",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"cron_log": true,
"log_type": "file",
"logfile": "/var/log/necen.data.surfsara.nl.log",
"logtimezone": "Europe/Amsterdam",
"loglevel": "0",
"logdateformat": "d/M/Y:H:i:s",
"objectstore_multibucket": {
"class": "\OC\Files\ObjectStore\Swift",
"arguments": {
"autocreate": true,
"username": "necen_xcloud",
"password": "blablabla",
"region": "RegionOne",
"url": "https://proxy.swift.surfsara.nl:5000/v2.0",
"tenantName": "necen_xcloud",
"serviceName": "Swift",
"urlType": "internal"
}
}
}
}

Are you using external storage, if yes which one: local/smb/sftp/...
yes, SWIFT

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser:

Operating system:

Logs

Web server error log

Web server error log

Insert your webserver log here

Nothing special

Nextcloud log (data/nextcloud.log)

Nextcloud log

Insert your Nextcloud log here

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[rontrompert]

Since SWIFT has a 5GB upload limit by default, I suspect that that is the issue. For uploading files larger than 5GB the files need to be chunked. Perhaps this is not implemented yet in the nextcloud I/F to SWIFT

[MorrisJobke]

cc @icewind1991

[rontrompert]

Since SWIFT has a 5GB upload limit by default, I suspect that that is the issue. For uploading files larger than 5GB the files need to be chunked. Perhaps this is not implemented yet in the nextcloud I/F to SWIFT

[rontrompert]

Maybe this is of interest to you: https://github.com/php-opencloud/openstack

[MorrisJobke]

Maybe this is of interest to you: https://github.com/php-opencloud/openstack

AFAIK this is already on the roadmap - cc @rullzer

[MorrisJobke]

I would say, this is a duplicate of #6106 - let's continue there.