Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't open / sync big files on external storage (webdav) #30282

Open
GAS85 opened this issue Dec 15, 2021 · 21 comments · Fixed by #33087
Open

Can't open / sync big files on external storage (webdav) #30282

GAS85 opened this issue Dec 15, 2021 · 21 comments · Fixed by #33087
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: external storage help wanted needs review Needs review to determine if still applicable performance 🚀

Comments

@GAS85
Copy link

GAS85 commented Dec 15, 2021

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce

  1. Add webdav external storage
  2. Tries to open big file there (e.g. 800 MB) or Sync folder via client
  3. See an error
  4. Check the logs and see:
GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi

Seems it is related to #30262 and #29969 with #26071
5. Use Workaround as per #26071 (comment)
6. See that Web UI is now opening, but with a huge delay (whole file being downloaded), but client sync does not work still.

Expected behaviour

Do not download files (as per #30262) when folder simply opened, use cache instead.
Do not download files when properties only needed (last modification date, or size)

Actual behaviour

External Storage could not be synchronized, even via Desktop Client it simply does not open and not synced any more.
All files downloaded to /tmp or RAM and causing timeout.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache/2.4.51

Database: 10.3.32-MariaDB

PHP version: 7.4.26

Nextcloud version: 22.2.3.0

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: Official

Signing status:

Signing status
No errors have been found.

List of activated apps:

App list
Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - admin_audit: 1.12.0
  - audioplayer: 3.2.2
  - bruteforcesettings: 2.2.0
  - calendar: 3.0.0
  - camerarawpreviews: 0.7.15
  - checksum: 1.1.2
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.6
  - contactsinteraction: 1.3.0
  - cospend: 1.4.3
  - dashboard: 7.2.0
  - data_request: 1.9.0
  - dav: 1.19.0
  - deck: 1.5.5
  - drawio: 1.0.1
  - external: 3.9.0
  - extract: 1.3.2
  - facerecognition: 0.8.5
  - federatedfilesharing: 1.12.0
  - files: 1.17.0
  - files_automatedtagging: 1.12.0
  - files_downloadactivity: 1.11.1
  - files_external: 1.13.0
  - files_mindmap: 0.0.25
  - files_pdfviewer: 2.3.1
  - files_retention: 1.11.1
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - files_videoplayer: 1.11.0
  - firstrunwizard: 2.11.0
  - forms: 2.4.0
  - gpxedit: 0.0.14
  - gpxpod: 4.3.0
  - integration_github: 1.0.2
  - integration_gitlab: 1.0.3
  - keeweb: 0.6.8
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - mail: 1.11.0
  - maps: 0.1.9
  - nextcloud_announcements: 1.11.0
  - notes: 4.2.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - ocdownloader: 1.7.12
  - password_policy: 1.12.0
  - phonetrack: 0.6.9
  - photos: 1.4.0
  - polls: 3.3.0
  - previewgenerator: 3.3.0
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - side_menu: 2.0.1
  - spreed: 12.1.2
  - support: 1.5.0
  - survey_client: 1.10.0
  - systemtags: 1.12.0
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - twofactor_totp: 6.1.0
  - unsplash: 1.2.4
  - updatenotification: 1.12.0
  - user_status: 1.2.0
  - video_converter: 1.0.3
  - viewer: 1.6.0
  - weather_status: 1.2.0
  - workflowengine: 2.4.0
Disabled:
  - circles
  - encryption
  - federation
  - files_accesscontrol
  - flowupload
  - gpxmotion
  - impersonate
  - integration_jira
  - radio
  - recognize
  - user_ldap
  - weather

Nextcloud configuration:

Config report
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "22.2.3.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 1.5
        },
        "default_phone_region": "DE",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "7be51475e95ea13b254cd880319d98ec",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1920,
        "preview_max_y": 1080,
        "jpeg_quality": 90,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "mail_sendmailmode": "smtp"
    }
}

Are you using external storage, if yes which one: local/webdav

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log
{"reqId":"3bdICeP14G6vPYqfCKqQ","level":3,"time":"2021-12-15T09:30:33+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":146,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/DAV.php","line":356,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":65,"function":"fopen","class":"OC\\Files\\Storage\\DAV","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":48,"function":"toTmpFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":269,"function":"getCachedFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":374,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":131,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":109,"function":"getActualValue","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":245,"function":"executeCheck","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":180,"function":"check","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":128,"function":"getMatchingOperations","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Operation.php","line":95,"function":"getFlows","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Listener/CacheListener.php","line":43,"function":"checkOperations","class":"OCA\\FilesAutomatedTagging\\Operation","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":75,"function":"handle","class":"OCA\\FilesAutomatedTagging\\Listener\\CacheListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":251,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":73,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":88,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":100,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","line":324,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"CustomMessage":"--"},"id":"61b9f1e0b44fa"}

Nextcloud Desktop Client log

Nextcloud Client log
||MagentaCloud|2|2|1638616226|_invalid_|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T13:59:38Z (last step: 157 msec, total: 2134 msec)
#=#=#=# Syncrun started 2021-12-15T13:59:40Z
#=#=#=#=# Propagation starts 2021-12-15T14:00:24Z (last step: 43733 msec, total: 43733 msec)
#=#=#=# Syncrun finished 2021-12-15T14:00:24Z (last step: 24 msec, total: 43757 msec)
#=#=#=# Syncrun started 2021-12-15T14:01:03Z
#=#=#=#=# Propagation starts 2021-12-15T14:01:03Z (last step: 371 msec, total: 371 msec)
||MagentaCloud|8|2|1638616226|61ab4ca2b0b9c|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T14:01:03Z (last step: 28 msec, total: 399 msec)
@GAS85 GAS85 added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Dec 15, 2021
@GAS85
Copy link
Author

GAS85 commented Dec 16, 2021

Also I can't open e.g. Videos from the external Storages via UI any more...

How it works before:

  1. Go to external storage
  2. Click on Video file
  3. Open immediately - nextlcoud works aka Proxy to the external Storage.

How it works now:

  1. Go to external storage
  2. Click on Video file
  3. See that Traffic goes up and memory usage grows:
    grafik
  4. Check /tmp Video files are there:
ls -lah /tmp | grep "www-data"
-rw-------  1 www-data www-data 800M Dec 16 08:52 oc_tmp_uLpKpT-.avi
-rw-------  1 www-data www-data  50M Dec 16 08:53 php0NKva5
-rw-------  1 www-data www-data 100M Dec 16 08:53 phpYRlgBW
-rw-------  1 www-data www-data 152M Dec 16 08:53 phptIAg9A
-rw-------  1 www-data www-data 200M Dec 16 08:53 phpwbuY14
  1. Nothing will be transferred till full Video downloaded from the external storage.
    grafik
  2. Close Video window, but traffic on backend will not be stopp till Video fully downloaded.
    grafik
  3. Nextcloud works aka caching contend provider to the external Storage.

@liejuntao001
Copy link

Add a data point.
I upgraded from 21.0.4 to 22.2.3.
The problem didn't happen in 21.0.4.
With 22.2.3 it's a new behavior that every time when user try to download a file from the webdav external storage, the file needs to be downloadeded to /tmp from the server side, then the web download could start.
If the server download could not finish within 30 seconds, the error happens.
A related, but might be different problem, the webdav external storage speed is degraded in 22 comparing to 21.0.4.
When do a full scan (occ scan ), it takes 10 minutes in 21 but 16 minutes in 22.

@liejuntao001
Copy link

This problem is still in 22.2.5.
When the user access a file and the WebDAV client is involved, Nextcloud would copy the whole file from remote to local /tmp folder. If the file is large, the delay could be very long and trigger curl time out.
This applies to the external webdav storage, and to the federation sharing.

@LukaszSellmann
Copy link

Problem still occurs in 23.0.2
External samba storage, folders with files larger than 512mb doesn't sync on entering them and shows as empty.
Command occ fieles:scan doesn't sync them as well.

@sk1p
Copy link

sk1p commented Jun 2, 2022

I'm pretty sure I've tracked down the issue. In #26266 the way Guzzle HTTP clients are constructed was changed from the default, which chose the handler based on the environment, to always use the curl handler. Just using the curl handler doesn't seem to support the "stream" argument like the default handler stack does - you need to somehow directly pass in the sink option, otherwise the response is written to a temporary file.

The following patch, on top of 23.0.3.2 works around this issue, although it also reverts the security improvements from said PR:

diff -ruN ./1/lib/private/Http/Client/ClientService.php ./0/lib/private/Http/Client/ClientService.php 
--- ./1/lib/private/Http/Client/ClientService.php   2022-03-21 13:02:41.000000000 +0000 
+++ ./0/lib/private/Http/Client/ClientService.php   2022-06-02 19:31:07.114812663 +0000 
@@ -68,11 +68,7 @@ 
         * @return Client 
         */ 
        public function newClient(): IClient { 
-               $handler = new CurlHandler(); 
-               $stack = HandlerStack::create($handler); 
-               $stack->push($this->dnsPinMiddleware->addDnsPinning()); 
- 
-               $client = new GuzzleClient(['handler' => $stack]); 
+               $client = new GuzzleClient(); 
 
                return new Client( 
                        $this->config,

@LukasReschke @rullzer could you have a look?

@sk1p sk1p mentioned this issue Jun 7, 2022
4 tasks
@kesselb
Copy link
Contributor

kesselb commented Jun 7, 2022

Good finding 👍

php-curl is a hard requirement for guzzle.

But you are right we are always using CurlHandler. The default implementation does a bit more: https://github.com/nextcloud/3rdparty/blob/ce3a3cedfd36dd914ec103a62b8a0f47e950a0e1/guzzlehttp/guzzle/src/Utils.php#L77-L106

Index: lib/private/Http/Client/ClientService.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lib/private/Http/Client/ClientService.php b/lib/private/Http/Client/ClientService.php
--- a/lib/private/Http/Client/ClientService.php	(revision e5309201f7479be1464fe10c71e4a039dddd26b9)
+++ b/lib/private/Http/Client/ClientService.php	(date 1654617117944)
@@ -63,8 +63,7 @@
 	 * @return Client
 	 */
 	public function newClient(): IClient {
-		$handler = new CurlHandler();
-		$stack = HandlerStack::create($handler);
+		$stack = HandlerStack::create();
 		$stack->push($this->dnsPinMiddleware->addDnsPinning());
 
 		$client = new GuzzleClient(['handler' => $stack]);

Does this work too?

@uellue
Copy link

uellue commented Jun 7, 2022

@kesselb Confirmed to work! 👍

@kesselb
Copy link
Contributor

kesselb commented Jun 8, 2022

Mind to sent a pull request?

@spyfly
Copy link

spyfly commented Jun 30, 2022

@kesselb This fixes the issue for Nextcloud 24.0.2 aswell 👍

I was previously experiencing timeouts when syncing larger files such as videos from/to an external WebDAV storage.

@szaimen
Copy link
Contributor

szaimen commented Jul 1, 2022

If no one wants to create the PR, shall I do it?

@szaimen szaimen added 2. developing Work in progress and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Jul 1, 2022
@szaimen
Copy link
Contributor

szaimen commented Jul 1, 2022

done

@spyfly
Copy link

spyfly commented Nov 4, 2022

Not fixed in the stable25 Branch: #34749

@GAS85
Copy link
Author

GAS85 commented Jul 24, 2023

Hey, I do the test and it is not really fixed.
There are to parts:

  • PRORFOUND will cause no download Mass periodical external Storage ALL files download recognize#116 (comment) any more - this is good.
  • Still can't download big files, NC will try to download file first and I must wait till it is 100% downloaded by the Server, but will hit Mem or/and Time limits, so that big Files sync not possible:
    Red arrow, when I click download on the UI, you see that Server tries to download this 1G file first and after some time I will see an error in UI and download stops without any Byte to be transferred to the UI.
    image

@GAS85
Copy link
Author

GAS85 commented Jul 25, 2023

@szaimen, @skjnldsv can we please reopen this issue? There is a comment and test in latest NC 25.0.9 that showing it is not fully implemented.

@kesselb
Copy link
Contributor

kesselb commented Jul 25, 2023

  1. Requests from external storage were not streamed anymore was fixed by fix external storages access #33087.
  2. The operation time issue was addressed by Add config variable for curl timeout #38292.

Please update to Nextcloud 27, set a higher timeout for dav storage, and create a new issue if the issue is still there.

@GAS85
Copy link
Author

GAS85 commented Jul 25, 2023

Thanks! NC 27.0.1 seems not mature enough for me now to make Prod update.
Nevertheless, when I look on a #33087 it was successfully backported to NC23 and NC24, but seems failed on NC25 - #34749 So from my point of view bug is still valid here and should not be closed till this time. Not sure if this fits to your process.

@kesselb
Copy link
Contributor

kesselb commented Jul 25, 2023

Oh, well spotted.

I don't understand the partial revert 🤷

@szaimen szaimen reopened this Jul 25, 2023
@szaimen szaimen added 0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback and removed 2. developing Work in progress 21-feedback labels Jul 25, 2023
@szaimen
Copy link
Contributor

szaimen commented Jul 25, 2023

Feel free to do another attempt on this with the same fix against master. Last time I tried the tests have been failing and I had no idea how to resolve them.

@s0nik42
Copy link

s0nik42 commented Jul 24, 2024

Hi, we still have the issue on 29.0.4. How can we help ?

@GAS85
Copy link
Author

GAS85 commented Aug 23, 2024

At least it is still an issue in 28.0.9.

@joshtrichards joshtrichards added needs review Needs review to determine if still applicable performance 🚀 labels Sep 5, 2024
@joshtrichards joshtrichards changed the title [NC 21] Can't open / sync big files on external storage (webdav) Can't open / sync big files on external storage (webdav) Sep 8, 2024
@loxK
Copy link

loxK commented Jan 2, 2025

I am currently facing the issue between two 30.0.4 servers. Has to increase the davstorage.request_timeout to 900 (default 30)

For reference, the option got introduced by #38292

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: external storage help wanted needs review Needs review to determine if still applicable performance 🚀
Projects
None yet
Development

Successfully merging a pull request may close this issue.