2.6.0: issues with HTTP/2

[fritzmg]

In 2.5.4, HTTP/2 was disabled in the ownCloud client due to issues within the Qt library. In 2.6.0 HTTP/2 was re-enabled again, as the issues were supposedly fixed.

However, when using the ownCloud client in version 2.6.0 now, syncing does not work with our ownCloud instance anymore, when there is a large number of files to be checked/synced. When using a PHP-FPM setup, we get the following error message within the client:

Server stopped accepting new streams before this stream was established

This error weirdly does not appear, when logging is enabled in the client. The client only times out after the same amount of time when that error would happen without logging. The log entries when logging is enabled are:

11-26 12:21:35:984 [ warning sync.networkjob ]:	Network job timeout QUrl("https://owncloud.example.org/remote.php/dav/files/ownclouduser/some/folder")
11-26 12:21:35:984 [ warning sync.networkjob ]:	QNetworkReply::OperationCanceledError "Connection timed out" QVariant(Invalid)
11-26 12:21:35:984 [ info sync.networkjob.lscol ]:	LSCOL of QUrl("https://owncloud.example.org/remote.php/dav/files/ownclouduser/some/folder") FINISHED WITH STATUS "OperationCanceledError Connection timed out"
11-26 12:21:35:984 [ warning sync.discovery ]:	LSCOL job error "Operation canceled" 0 QNetworkReply::OperationCanceledError
11-26 12:21:35:984 [ warning sync.discovery ]:	Server error in directory "some/folder" 0
11-26 12:21:35:984 [ info sync.engine ]:	Sync run took  328229 ms
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "STATUS:OK:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ debug sync.localdiscoverytracker ]	[ OCC::LocalDiscoveryTracker::slotSyncFinished ]:	sync failed, keeping last sync's local discovery path list
11-26 12:21:35:984 [ debug sync.networkjob ]	[ OCC::AbstractNetworkJob::slotFinished ]:	Network job OCC::LsColJob finished for "/some/folder"
11-26 12:21:35:984 [ info gui.folder ]:	Client version 2.6.0 (build 12644)  Qt 5.12.5  SSL  OpenSSL 1.1.1d  10 Sep 2019
11-26 12:21:35:984 [ warning gui.folder ]:	SyncEngine finished with ERROR
11-26 12:21:35:984 [ info gui.folder ]:	Folder sync result:  2
11-26 12:21:35:984 [ info gui.folder ]:	the last 1 syncs failed
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "STATUS:OK:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "UPDATE_VIEW:…\\ownCloud" to QLocalSocket(0x796f0e0)
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "UPDATE_VIEW:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ info gui.application ]:	Sync state changed for folder  "https://owncloud.example.org/remote.php/dav/files/ownclouduser/" :  "Error"

On the server side, this is the only error that coincides:

[Tue Nov 26 17:01:19.811110 2019] [proxy_fcgi:error] [pid 24501] (32)Broken pipe: [client 212.186.187.58:48992] AH01075: Error dispatching request to : (passing brigade to output filters)

There are no PHP, PHP-FPM or MySQL errors present in the respective error logs otherwise.

Since the source error message ("Server stopped accepting new streams before this stream was established") comes from within the Qt library here, it looks like it has something to do with HTTP/2 again.

And sure enough, after disabling HTTP/2 on the server side, no synchronisation errors happened any more.

Server configuration

Operating system:
Ubuntu 16.04.6

Web server:
Apache 2.4.18
nginx 1.16.1.3

Database:
MySQL 5.7.28

PHP version:
7.3.12

ownCloud version:
10.3.1

Client configuration

Client version:
2.6.0

Operating system:
Windows

OS language:
English

Installation path of client:
C:\Program Files (x86)\ownCloud

[guruz]

@fritzmg Thank you for the report. This sounds unfortunate.

@ckamm Do you think this could be https://bugreports.qt.io/browse/QTBUG-77852 ?

[guruz]

@fritzmg Just wondering which Apache version exactly you use?
And which model in Apache (worker, MPM, ...)
I'm also curious about the FPM version

[felixboehm]

ownClouds running always with mod prefork, which doesn‘t support http/2.
It would be interesting to have a test with php fpm using http/2, also in terms of speed ...

[fritzmg]

@fritzmg Just wondering which Apache version exactly you use?

2.4.18

However, the issue is also present when using nginx only (version 1.16.1)

And which model in Apache (worker, MPM, ...)

Hm, not sure, how do I check that?

I'm also curious about the FPM version

Not sure what you mean there - isn't that just the PHP version?

[michaelstingl]

@TheOneRing please disable again for the 2.6.1

[fritzmg]

Before you revert anything: can anybody else confirm the issue? This might still be a server configuration issue. As far as I know, I have done every conceivable and recommended server configuration - but there might still be something, that needs to be done. I just need to find out what it is.

[jnweiger]

I haven't tried HTTP2 myself, but we collected some info. It probably requires php-FPM in any case, which is not our recommended setup. There are two use cases

• the ownCloud server itself does the FPM so that HTTP2 can work,
• a reverse proxy (e.g. nginx) does HTTP2 -- in this case the issue can be considered independant of support in the ownCloud server.

[fritzmg]

It probably requires php-FPM in any case, which is not our recommended setup.

Why is that? The official documentation does not mention that anywhere, at least as far as I see. PHP-FPM should always be the recommended way of running PHP.

[michaelstingl]

@fritzmg here you can find some background: owncloud/core#30725

[fritzmg]

@michaelstingl interesting, thanks. But still, the issues I am having also happen with a pure nginx + PHP-FPM setup, while owncloud/core#30725 mostly talks about Apache support of HTTP/2.

[michaelstingl]

@fritzmg for the upcoming 2.6.1 release, we removed HTTP/2 support, but you can enable it for testing using the environment variable OWNCLOUD_HTTP2_ENABLED . No ETA a yet to bring it back…

[guruz]

@tessus Have you seen this in your testing?

[tessus]

@guruz no, but I didn't do extensive testing out of fear. IMO a lot of VFS issues bled over to the basic sync code for some reason and the entire client is less stable than before.

Currently I'm still using an ancient version for 2 reasons: h2 works, no VFS code:

Version 2.4.3 (build 10347).
Built from Git revision 560555 on Aug 10 2018, 21:27:04 using Qt 5.10.1, OpenSSL 1.0.2n 7 Dec 2017

But let me comment on a few things here, because I have tthe feeling that some people do not understand h2, backend, and fpm.

• h2 REQUIRES a worker or event MPM
• worker or event REQUIRES php-fpm

h2 uses a thread model, which will break when using a prefork MPM on the web server, which is process based. Furthermore a process based server has several other disadvantages as well: one php version per server, php caches (apcu, opcache, ...) will be lost during httpd server restart, ...
php-fpm is part of PHP, which means that the version of fpm is the same as the PHP version.
When using a thread based httpd MPM, php can't be used as a module, unless you use ZTS, which has a whole set of painful side effects - "opening a can of worms" comes to mind.

Any sane server environment these days uses a thread based model, h2, and php-fpm (and maybe a reverse proxy somwhere in the mix). Everything else is less performant, more complicated to maintain, and less secure.

As a side note, I find it really disconcerting that h2 is still a problem and that you've decided to deactivate it again instead of making it the highest priority. There are server admins out there who have disabled http/1.0 and http/1.1 completely. The reasons should be obvious, but I will not start a h2 promotion thread here.
I'm already at the lookout for a stable 3rd party client that uses WebDAV and h2 for syncing. All the other features are useless to me anyway, since I am not using macOS Finder (so I can't right-click for the share context menu), nor am I interested in VFS. And to be honest I don't trust the newer ownCloud or Nextcloud clients enough to keep my data safe. (Please forgive me, but that's my opinion.)

P.S.: I'm more than willing to re-evaluate the ownCloud client again in the future. I'm not saying that I'm gonna turn my back on it forever. It's just that the current situation doesn't create much confidence.

[dipyalov]

Have the same issue after 2.6.0
Plus all of my virtual files are gone and I'm worried that ownCloud may decide to sync-delete them from server.

[tessus]

I'm very puzzled as to why this issue has been closed. The issues with H2 haven't been fixed. Instead H2 was deactivated once again.

I honestly don't understand why you don't stop any other development to get the most fundamental issue fixed. Nothing is more important than the transport of data. I'm not making this up. It's the 1st layer of the application layer (OSI).
If this doesn't work, the rest won't work either.

Anyway, maybe in 5 or 10 years H2 will work. Until then I'll use a 5 year old client that actually does work with H2.

[jnweiger]

Sorry to hear so much frustration. Yes priorities appear to differ here. Not sure how to actually make progress...
Maybe the HTTP2 issue should also be discussed with the server team?
As for the client side, do not despair. I'll try to spark a constructive discussion in #7626 which was scheduled for 2.6.2 already in December.

[tessus]

@jnweiger IMO there is no issue with H2 on the server unless the server admins use a wrong nghttp2 version. Apache httpd depends on nghttp2 and if that lib is broken, h2 can't work. Stefan Eissing, the dev of mod_h2 in httpd has done a great job making the h2 module stable over the past few years.

Of course there's a chance that it is a combination of server/client issues that manifests itself in the client. But let's put it that way: I've been using H2 for a long time and I haven't had any issues with any other client.

@ho4ho the QT implemrntation of H2 has been broken since forever. There's always: "ah, there's this bug, which will be fixed in QT x.x.x", then it takes forever to switch the client to use that QT version and then it starts all over. If QT is unable to do H2 properly, a different approach must be taken. e.g. using the nghttp2 lib directly and use a wrapper to use from the oc client.

With all due respect, but after a few years of trying to get H2 working without success, somebody must accept that the current approach is not working.
In 2020 one should be able to expect H2 to be working flawlessly.