Make App Store timeouts longer or configurable

[canoine]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

As my internet connection is a joke, most of the times, curl stops on error 28 :

Aug 28 17:57:11 xxx nextcloud[9192]: {"reqId":"ZOzDmyKEzRATGUE9FKaEWwAAAE8","level":2,"time":"2023-08-28T17:57:11+02:00","remoteAddr":"90.62.30.145","user":"admin","app":"appstoreFetcher","method":"GET","url":"/index.php/settings/apps","message":"Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 4722688 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0","version":"26.0.5.1","data":{"app":"appstoreFetcher"}}

or
Aug 28 18:32:06 xxx nextcloud[11122]: {"reqId":"ZOzLjSEcSVqe3qzIFqTXUwAAAAY","level":3,"time":"2023-08-28T18:32:06+02:00","remoteAddr":"90.62.30.145","user":"admin","app":"settings","method":"POST","url":"/index.php/settings/apps/enable","message":"{\"Exception\":\"GuzzleHttp\\\\Exception\\\\ConnectException\",\"Message\":\"cURL error 28: Operation timed out after 120000 milliseconds with 2322560 out of 5238706 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://objects.githubusercontent.com/github-production-release-asset-2e65be/187186203/e96bc648-7e72-4612-90e0-3524b7ee8e12?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230828%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230828T162757Z&X-Amz-Expires=300&X-Amz-Signature=65d518673cf3a25d9191d0dfefd560962fd48e91b23acafaaf0807e5ac0a0d2c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=187186203&response-content-disposition=attachment%3B%20filename%3DCookbook-0.10.2.tar.gz&response-content-type=application%2Foctet-stream\",\"Code\":0,\"Trace\":[{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php\",\"line\":158,\"function\":\"createRejection\",\"class\":\"GuzzleHttp\\\\Handler\\\\CurlFactory\",\"type\":\"::\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php\",\"line\":110,\"function\":\"finishError\",\"class\":\"GuzzleHttp\\\\Handler\\\\CurlFactory\",\"type\":\"::\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php\",\"line\":47,\"function\":\"finish\",\"class\":\"GuzzleHttp\\\\Handler\\\\CurlFactory\",\"type\":\"::\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php\",\"line\":150,\"function\":\"__invoke\",\"class\":\"GuzzleHttp\\\\Handler\\\\CurlHandler\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php\",\"line\":35,\"function\":\"OC\\\\Http\\\\Client\\\\{closure}\",\"class\":\"OC\\\\Http\\\\Client\\\\DnsPinMiddleware\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php\",\"line\":31,\"function\":\"__invoke\",\"class\":\"GuzzleHttp\\\\PrepareBodyMiddleware\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php\",\"line\":71,\"function\":\"GuzzleHttp\\\\{closure}\",\"class\":\"GuzzleHttp\\\\Middleware\",\"type\":\"::\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php\",\"line\":107,\"function\":\"__invoke\",\"class\":\"GuzzleHttp\\\\RedirectMiddleware\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php\",\"line\":73,\"function\":\"checkRedirect\",\"class\":\"GuzzleHttp\\\\RedirectMiddleware\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/FulfilledPromise.php\",\"line\":41,\"function\":\"GuzzleHttp\\\\{closure}\",\"class\":\"GuzzleHttp\\\\RedirectMiddleware\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php\",\"line\":48,\"function\":\"GuzzleHttp\\\\Promise\\\\{closure}\",\"class\":\"GuzzleHttp\\\\Promise\\\\FulfilledPromise\",\"type\":\"::\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php\",\"line\":248,\"function\":\"run\",\"class\":\"GuzzleHttp\\\\Promise\\\\TaskQueue\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php\",\"line\":224,\"function\":\"invokeWaitFn\",\"class\":\"GuzzleHttp\\\\Promise\\\\Promise\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php\",\"line\":269,\"function\":\"waitIfPending\",\"class\":\"GuzzleHttp\\\\Promise\\\\Promise\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php\",\"line\":226,\"function\":\"invokeWaitList\",\"class\":\"GuzzleHttp\\\\Promise\\\\Promise\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php\",\"line\":62,\"function\":\"waitIfPending\",\"class\":\"GuzzleHttp\\\\Promise\\\\Promise\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php\",\"line\":187,\"function\":\"wait\",\"class\":\"GuzzleHttp\\\\Promise\\\\Promise\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/Http/Client/Client.php\",\"line\":226,\"function\":\"request\",\"class\":\"GuzzleHttp\\\\Client\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/Installer.php\",\"line\":295,\"function\":\"get\",\"class\":\"OC\\\\Http\\\\Client\\\\Client\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/apps/settings/lib/Controller/AppSettingsController.php\",\"line\":448,\"function\":\"downloadApp\",\"class\":\"OC\\\\Installer\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":230,\"function\":\"enableApps\",\"class\":\"OCA\\\\Settings\\\\Controller\\\\AppSettingsController\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":137,\"function\":\"executeController\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/AppFramework/App.php\",\"line\":183,\"function\":\"dispatch\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/private/Route/Router.php\",\"line\":315,\"function\":\"main\",\"class\":\"OC\\\\AppFramework\\\\App\",\"type\":\"::\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/lib/base.php\",\"line\":1065,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\"},{\"file\":\"/var/www/localhost/htdocs/nextcloud/index.php\",\"line\":36,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\"}],\"File\":\"/var/www/localhost/htdocs/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php\",\"Line\":210,\"message\":\"could not enable apps\",\"exception\":{},\"CustomMessage\":\"could not enable apps\"}","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0","version":"26.0.5.1"}

So, most of the times, the app store is missing, or I can't update my apps.

Yes, my connection is slow, but it works. So the downloads are in progress, not stalled. Curl should be more patient, I think.
With his ugly workaround, I can do my updates :

lib/private/App/AppStore/Fetcher/Fetcher.php: // 'timeout' => 60,
lib/private/App/AppStore/Fetcher/Fetcher.php: 'timeout' => 600,
lib/private/Preview/Imaginary.php: // 'timeout' => 120,
lib/private/Preview/Imaginary.php: 'timeout' => 600,

Steps to reproduce

1. Live far from the civilized world
2. Try to update your nextcloud apps

Expected behavior

After a consequent amount of patience and coffee, apps are updated.

Installation method

Other Community project

Nextcloud Server version

26

Operating system

Other

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

[joshtrichards]

I think this is more an enhancement request than a bug. :-)

I don't know if we can bump these up too high. There could be some side effects. But it might be okay.

Another possibility, which would take a bit more work but would also be safer, is to expose the curl timeout as a configuration option. We already expose timeout parameters for a lot of the other services.

[canoine]

I think this is more an enhancement request than a bug. :-)

Well, okay. :)

I don't know if we can bump these up too high. There could be some side effects. But it might be okay.

I don't know what would be the side effects of these huge timeouts. I put these as a quick and dirty workaround, that's it.
I don't know how curl works, neither. But as I see that a SFTP transfer goes on as long as the data comes, even slowly, perhaps there is an option for curl to adopt the same behaviour ?

Another possibility, which would take a bit more work but would also be safer, is to expose the curl timeout as a configuration option. We already expose timeout parameters for a lot of the other services.

If the amount of work is acceptable, and the result sufficiently safe for all, That seems a good solution.

[joshtrichards]

The Server Updater is also presumably impacted in a similar way as the App Store.

[bernd-wechner]

Another possibility, which would take a bit more work but would also be safer, is to expose the curl timeout as a configuration option. We already expose timeout parameters for a lot of the other services.

Crucial IMHO to those of us affected by this (server responses taking longer than the nominal timeout). It was changed in: #16972 with the comment "A default timeout of 30 seconds should cover the 99% case. If a job need specific longer time it should set that." which alas is not so simple. To be sure I updated it and it works, for our local need, but now that file fails its signature test (is detected by Nextcloud as modified code and reported as such).

The hash no differs from that in core/signature.json. And it is (near) impossible to fix that, changing the hash only destroys the integrity of that file which has its own internally recorded signature signed with a certificate. TO with removing such a local mod from the error list would requite regenerating an integral core/signature.json which (I presume and expect) would require a private key to do.

The doc only says ["Please don’t modify the mentioned signature.json itself."](https://docs.nextcloud.com/server/27/admin_manual /issues/code_signing.html#errors) with no explanation of why not. Why not is, simply that it's not so simple, that the file is signed certified internally, and any change to it just produces another reported error. The only way to remove it then would be the sledge hammer approach of modifying lib/private/IntegrityCheck/Checker.php to exempt this file explicitly, and consequently probably, also exempting itself explicitly from the hash check.

A load of rigmarole and not a simple case of "If a job need specific longer time it should set that.".

To wit, this should most definitely be a setting available in config.php! Given it is something that may need tuning to local network performance conditions -- i.e. some of us do a clean install and find no App Store and scratch our heads and waste a load of time, to get to this point right here all of which would shrink if a) there was a clear message on the App store that the timeout had been exceeded with a hint to set it in config.pp and b) it could be set in config.php.

[TobiPeterG]

I'm affected by this as well and another user posted about this only a few days ago:
https://help.nextcloud.com/t/how-to-change-the-appstore-fetcher-timeout/208720

Here is my log:
Could not connect to appstore: cURL error 28: Operation timed out after 60001 milliseconds with 420554 out of 7026000 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

However, accessing the page from my web browser loads it instantly, so I don't get why nextcloud is failing to.

I have nextcloud AIO in version Nextcloud Hub 8 (29.0.8)

[joshtrichards]

However, accessing the page from my web browser loads it instantly, so I don't get why nextcloud is failing to.

Then it sounds like your need isn't a change in the timeout option. Can you test from within your container? This sounds like some sort of outbound connectivity problem within your Docker environment that a timeout change just covers up rather than fixes.

[detcordtech]

Finally landed here after 3 days of trying to figure out why the appstore won't load on multiple VMs and different versions of nextcloud... Did various debugging steps on these two:

• v30.0.2
• v29.0.12

The URL https://apps.nextcloud.com/api/v1/apps.json takes well over 1 minute for me to load everywhere, cURL via linux terminal, browser and for nextcloud too, but my connection is not slow.

I'm seeing the same error message as TobiPeterG showing timeout before the download finished.

Please consider treating this ticket as a bug, because in certain regions, even with 1 Gbps fiber link, for some reason connection to this URL is slow and does not finish loading within 60 seconds. The timeout is not long enough.

At first I was convinced something is broken in the last few releases because app store wouldn't load correctly.

Screenshot below shows ~500Mbps download speed via speedtest-cli while nextcloud apps.json download took 10.5 minutes to complete. Not sure why this is slower for me here.

Ridiculous as it may sound, "15 minute timeout for a 15 MB file?", I like to think - better slow for once, cached for later, than completely broken and impossible to load/seemingly broken.

Timeouts should be configurable via config file instead of having to modify code to get around situations like these.

Until then I will modify php files as mentioned in the first post to hopefully get around this.

Thanks!

[JMarcosHP]

Finally landed here after 3 days of trying to figure out why the appstore won't load on multiple VMs and different versions of nextcloud... Did various debugging steps on these two:

• v30.0.2
• v29.0.12

The URL https://apps.nextcloud.com/api/v1/apps.json takes well over 1 minute for me to load everywhere, cURL via linux terminal, browser and for nextcloud too, but my connection is not slow.

I'm seeing the same error message as TobiPeterG showing timeout before the download finished.

Please consider treating this ticket as a bug, because in certain regions, even with 1 Gbps fiber link, for some reason connection to this URL is slow and does not finish loading within 60 seconds. The timeout is not long enough.

At first I was convinced something is broken in the last few releases because app store wouldn't load correctly.

Screenshot below shows ~500Mbps download speed via speedtest-cli while nextcloud apps.json download took 10.5 minutes to complete. Not sure why this is slower for me here.

Ridiculous as it may sound, "15 minute timeout for a 15 MB file?", I like to think - better slow for once, cached for later, than completely broken and impossible to load/seemingly broken.

Timeouts should be configurable via config file instead of having to modify code to get around situations like these.

Until then I will modify php files as mentioned in the first post to hopefully get around this.

Thanks!

I have this issue too, just upgraded to Nextcloud 30 using the Mastercontainer and now the Appstore is not loading anymore.

[SystemKeeper]

@detcordtech Would you be able to retry your tests and post the results please?