No sync with Baikal

[Tom-Kov]

After installing rmcarddav (on Linux 5.6.14, fedora 31, on x86_64) I could well configure a connection to my Baikal carddav server - but no contacts got downloaded. It shows the proper name of the addressbook, as configured in Baikal, but no contacts. In the syslog I only find a message that "BACKEND: (list_records_sync_collection) An error (status 401) occured".
When I create a new contact in rmcarddav, it gets properly synced into Baikal, as well as any changes and the deletion of this new contact. Only the contacts already existing in Baikal as well as contacts newly created in other ways do not show up.
Using roundcube 1.4.6, rmcarddav 3.0.3, Baikal 0.7.1, MariaDB 10.3.22.

[mstilkerich]

Hello,

if you are willing to try unreleased software, you can try the version 4 development branch. I do not have a Baikal server to test, but Baikal uses SabreDAV as does Nextcloud, which is known to work. If you find issues, I would appreciate feedback and I'll see to fix it.

See installation instructions here: https://github.com/blind-coder/rcmcarddav/tree/v4

[Tom-Kov]

Hi,

thanks. No problem hacking :-). I have installed the version 4 dvlp branch - and when trying to configure the CardDAV server, after "save" the browser screen gets empty and in the php log, the following is found:

[29-Jun-2020 16:59:58 Europe/Prague] PHP Fatal error:  Uncaught Exception: Expected Multistatus HTTP request was not successful (401 Unauthorized): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:sabredav-version>4.1.0</s:sabredav-version>
  <s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
  <s:message>No 'Authorization: Digest' header found. Either the client didn't send one, or the server is misconfigured. Login was needed for privilege: {DAV:}read on </s:message>
</d:error>
 in /var/www/html/roundcubemail-1.4.6/vendor/mstilkerich/carddavclient/src/CardDavClient.php:347
Stack trace:
#0 /var/www/html/roundcubemail-1.4.6/vendor/mstilkerich/carddavclient/src/CardDavClient.php(355): MStilkerich\CardDavClient\CardDavClient::assertHttpStatus(Object(GuzzleHttp\Psr7\Response), 207, 207, 'Expected Multis...')
#1 /var/www/html/roundcubemail-1.4.6/vendor/mstilkerich/carddavclient/src/CardDavClient.php(303): MStilkerich\CardDavClient\CardDavClient::checkAndParseXMLMultistatus(Object(GuzzleHttp\Psr7\Respon in /var/www/html/roundcubemail-1.4.6/vendor/mstilkerich/carddavclient/src/CardDavClient.php on line 347

Liebe Grüße
--- Tom

[mstilkerich]

Hi,

thanks for trying! It looks like an authentication issue (I'll take care of the unhandled exception though, it should not result in a white page).

• You are sure that username / password you provided are correct?
• Can you enable debugging in config.inc.php (see config.inc.php.dist for the two relevant settings, set both to DEBUG), and provide the content of carddav_http.log? Please scan the content first for confidential data and redact, but please don't remove too much as this issue seems to be related with authentication.
• Which authentication type have you configured in Baikal?

[mstilkerich]

Nevermind it's the digest authentication that is not working. I now have a Baikal server and can reproduce the problem.

[Tom-Kov]

Perfect, you are fast :-), even before I could respond.
Yes, I am using digest authentication. I assume you do not need the debug output now?

[mstilkerich]

You can try composer.phar update. It should update carddavclient to v0.1.1 which works for me.

You need the php-curl extension installed for any auth mechanism other than Basic to work.

[Tom-Kov]

I am using the php-curl extension, yes.
I have made the composer update now. In the configuration, I do not get the blank screen any more and all of the addressbooks of the given principal are now found and listed (providing http://host/card.php/addressbooks/principal-name/addressbook/ now lists all of the addressbooks of the principal, as if stating http://host/card.php/principals/principal-name/).

But, the addressbooks are only listed, but no contacts are imported. PHP throws an exception (apparently for each contact in the addressbook) -

[03-Jul-2020 17:54:50 Europe/Prague] Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /var/www/html/baikal/vendor/sabre/xml/lib/Service.php:122

Stack trace:

#0 /var/www/html/baikal/vendor/sabre/dav/lib/DAV/CorePlugin.php(680): Sabre\Xml\Service->parse('', '/card.php/addre...', NULL)

#1 /var/www/html/baikal/vendor/sabre/event/lib/WildcardEmitterTrait.php(89): Sabre\DAV\CorePlugin->httpReport(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))

#2 /var/www/html/baikal/vendor/sabre/dav/lib/DAV/Server.php(470): Sabre\DAV\Server->emit('method:REPORT', Array)

#3 /var/www/html/baikal/vendor/sabre/dav/lib/DAV/Server.php(251): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))

#4 /var/www/html/baikal/vendor/sabre/dav/lib/DAV/Server.php(319): Sabre\DAV\Server->start()

#5 /var/www/html/baikal/Core/Frameworks/Baikal/Core/Server.php(119): Sabre\DAV\Server->exec()

#6 /var/www/html/baikal/html/card.php(74): Baikal\Core\Server->start()

#7 {main}

[mstilkerich]

Hey we're getting there ;-) The exceptions you show are actually thrown on the Baikal server side, not at the roundcube side. It sounds to me like the request body or an element of it is empty, but I have no explanation for that.

Concerning the addressbooks discovery: You need to provide the plugin with enough information so it can find the addressbook home / principal URI of the user. If discovery is set up, a hostname is sufficient. But even if you provide directly the URL to an addressbook, it will ask the server for all addressbooks of that principal. If you don't want some of the addressbooks, you can deactivate them in the settings of the plugin.

Back to the issue: Unfortunately I cannot reproduce the issue (tested using Baikal 0.7.1 with an addressbook > 100 contacts and Digest auth), so the log files might be helpful after all. You find them in roundcube/logs/: carddav.log, carddav_http.log, errors.log. If you don't need the log contents, best clear them, then run a resync from the settings of the plugin and take the log files afterwards. With Digest authentication, there should be no Authorization headers in the logfile, but if the sync works your contact vcards would be contained.

[freeman1doma]

Have the same issue with baikal 0.7.1(digest auth scheme), roundcube 1.4.4 and rcmcarddav 3.0.3 (freebsd 12.1 amd64, php 7.2), all address-books have added but no see any contacts. After upgrade rcmcarddav to v4 dev the same issue present and if I want save new contact in book have ondisplay error: "Single-row query (SELECT id FROM carddav_contacts WHERE cuid = 'sabre-vobject-972bd6c1-e45e-4947-8289-ed34223195a3') without result". And from carddav.log:

[07-Jul-2020 21:09:41 +0000]: <hc1ocqls> sync-collection REPORT produced exception{"exception":{}} [07-Jul-2020 21:09:41 +0000]: <hc1ocqls> Errors occurred during the refresh of addressbook 8: Exception: Expected Multistatus HTTP request was not successful (500 Internal Server Error): <?xml version="1.0" encoding="utf-8"?> <d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns"> <s:sabredav-version>4.1.0</s:sabredav-version> <s:exception>Sabre\Xml\ParseException</s:exception> <s:message>The input element to parse is empty. Do not attempt to parse</s:message> </d:error> in /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php:354 Stack trace: #0 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(362): MStilkerich\CardDavClient\CardDavClient::assertHttpStatus(Object(GuzzleHttp\Psr7\Response), 207, 207, 'Expected Multis...') #1 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(279): MStilkerich\CardDavClient\CardDavClient::checkAndParseXMLMultistatus(Object(GuzzleHttp\Psr7\Response)) #2 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(253): MStilkerich\CardDavClient\CardDavClient->multiGet('https://cal.agr...', Array, Array) #3 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(102): MStilkerich\CardDavClient\Services\Sync->multiGetChanges(Object(MStilkerich\CardDavClient\CardDavClient), Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavClient\Services\SyncResult), Array) #4 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(520): MStilkerich\CardDavClient\Services\Sync->synchronize(Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavAddressbook4Roundcube\SyncHandlerRoundcube), Array, '') #5 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(361): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->resync(true) #6 /usr/local/www/roundcube/plugins/carddav/carddav.php(367): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->__construct(Object(rcube_db_mysql), '8', Object(carddav)) #7 /usr/local/www/roundcube/program/lib/Roundcube/rcube_plugin_api.php(457): carddav->getAddressbook(Array) #8 /usr/local/www/roundcube/program/include/rcmail.php(226): rcube_plugin_api->exec_hook('addressbook_get', Array) #9 /usr/local/www/roundcube/program/steps/addressbook/func.inc(152): rcmail->get_address_book('carddav_8', true) #10 /usr/local/www/roundcube/program/steps/addressbook/save.inc(20): rcmail_contact_source('carddav_8', true, true) #11 /usr/local/www/roundcube/index.php(306): include_once('/usr/local/www/...') #12 {main} [07-Jul-2020 21:09:42 +0000]: <hc1ocqls> sync-collection REPORT produced exception{"exception":{}} [07-Jul-2020 21:09:42 +0000]: <hc1ocqls> Errors occurred during the refresh of addressbook 8: Exception: Expected Multistatus HTTP request was not successful (500 Internal Server Error): <?xml version="1.0" encoding="utf-8"?> <d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns"> <s:sabredav-version>4.1.0</s:sabredav-version> <s:exception>Sabre\Xml\ParseException</s:exception> <s:message>The input element to parse is empty. Do not attempt to parse</s:message> </d:error> in /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php:354 Stack trace: #0 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(362): MStilkerich\CardDavClient\CardDavClient::assertHttpStatus(Object(GuzzleHttp\Psr7\Response), 207, 207, 'Expected Multis...') #1 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(279): MStilkerich\CardDavClient\CardDavClient::checkAndParseXMLMultistatus(Object(GuzzleHttp\Psr7\Response)) #2 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(253): MStilkerich\CardDavClient\CardDavClient->multiGet('https://cal.agr...', Array, Array) #3 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(102): MStilkerich\CardDavClient\Services\Sync->multiGetChanges(Object(MStilkerich\CardDavClient\CardDavClient), Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavClient\Services\SyncResult), Array) #4 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(520): MStilkerich\CardDavClient\Services\Sync->synchronize(Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavAddressbook4Roundcube\SyncHandlerRoundcube), Array, '') #5 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(1501): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->resync() #6 /usr/local/www/roundcube/program/steps/addressbook/save.inc(225): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->insert(Array) #7 /usr/local/www/roundcube/index.php(306): include_once('/usr/local/www/...') #8 {main}

At the same time thunderbird with cardbook plugin and android with davx5 worked like a charm with baikal.

[freeman1doma]

And if refresh address-book have this error in log:

[07-Jul-2020 21:22:28 +0000]: <ld0omijh> sync-collection REPORT produced exception{"exception":{}} [07-Jul-2020 21:22:28 +0000]: <ld0omijh> Errors occurred during the refresh of addressbook 8: Exception: Expected Multistatus HTTP request was not successful (500 Internal Server Error): <?xml version="1.0" encoding="utf-8"?> <d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns"> <s:sabredav-version>4.1.0</s:sabredav-version> <s:exception>Sabre\Xml\ParseException</s:exception> <s:message>The input element to parse is empty. Do not attempt to parse</s:message> </d:error> in /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php:354 Stack trace: #0 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(362): MStilkerich\CardDavClient\CardDavClient::assertHttpStatus(Object(GuzzleHttp\Psr7\Response), 207, 207, 'Expected Multis...') #1 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/CardDavClient.php(279): MStilkerich\CardDavClient\CardDavClient::checkAndParseXMLMultistatus(Object(GuzzleHttp\Psr7\Response)) #2 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(253): MStilkerich\CardDavClient\CardDavClient->multiGet('https://cal.agr...', Array, Array) #3 /usr/local/www/roundcube/vendor/mstilkerich/carddavclient/src/Services/Sync.php(102): MStilkerich\CardDavClient\Services\Sync->multiGetChanges(Object(MStilkerich\CardDavClient\CardDavClient), Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavClient\Services\SyncResult), Array) #4 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(520): MStilkerich\CardDavClient\Services\Sync->synchronize(Object(MStilkerich\CardDavClient\AddressbookCollection), Object(MStilkerich\CardDavAddressbook4Roundcube\SyncHandlerRoundcube), Array, '') #5 /usr/local/www/roundcube/plugins/carddav/src/Addressbook.php(361): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->resync(true) #6 /usr/local/www/roundcube/plugins/carddav/carddav.php(367): MStilkerich\CardDavAddressbook4Roundcube\Addressbook->__construct(Object(rcube_db_mysql), '8', Object(carddav)) #7 /usr/local/www/roundcube/program/lib/Roundcube/rcube_plugin_api.php(457): carddav->getAddressbook(Array) #8 /usr/local/www/roundcube/program/include/rcmail.php(226): rcube_plugin_api->exec_hook('addressbook_get', Array) #9 /usr/local/www/roundcube/program/steps/addressbook/func.inc(152): rcmail->get_address_book('carddav_8', false) #10 /usr/local/www/roundcube/program/steps/addressbook/list.inc(47): rcmail_contact_source('carddav_8', true) #11 /usr/local/www/roundcube/index.php(306): include_once('/usr/local/www/...') #12 {main}

[mstilkerich]

Hello,

thanks for the feedback. Could you please set the two loglevels to debug in plugins/carddav/config.inc.php and provide both carddav.log and carddav_http.log?

It seems that the multiget report is considered malformed / empty by the server (this should be visible in the HTTP log). The multiget is the second report in a sync run, and it is only called if the server reported changes for the first report. This would rule out the authentication mechanism as the issue.
The issue with the database query failing is essentially for the same reason. The plugin stores the card to the server and then attempts a resync to get the card back from the server. If the sync reports don't work, this error will result. Did the added card appear with other clients?

[mstilkerich]

@kovarto @freeman1doma: Anyone still interested in this issue? I would need a log file, because in my development environment everything works just fine with Baikal 0.7.1 + Digest Auth. I would be particularly interested in the multiget request sent to the server, that seems to raise an exception in Baikal.

[TravisNice]

@mstilkerich I'm having this issue. I'm using Baikal 0.7.1 + Digest Auth, Roundcube 4 - dev from your suggestion above, php7.4, apache 2.4.38. I've add the log files you asked for above ...

Interesting behaviour: Roundcube's contacts won't load from CardDav, however; it will create a new contact which then successfully sync's with my other devices. I can edit the contact on my iPhone which sync's to my desktop, but it won't sync back to Roundcube's contact.
carddav_http.log
carddav.log

[mstilkerich]

Thanks. The authentication is working fine, hence it is not surprising that storing a contact works. You won't see it in roundcube though, because all modifications are done to the server and then need to sync back to roundcube's database. And the syncing part is not working.

The sync consists of two stages, each has a fast and a slow path.

1. Determine what changed on the server (deleted / added / changed contacts)

• Fast path: WebDAV servers provide a dedicated sync-collection REPORT to report the changes. This report fails.
• Fallback path: Ask the server on the state on all objects in the addressbook. This query works in your case.

2. Download the changes

• Fast path: CardDAV servers provide a dedicated multiget REPORT to download multiple address objects with one request. This report fails again.
• Fallback path: Fetch each changed object with an individual GET request. This fallback did not work because the client library aborts when the multiget report fails with a server error.

Point 1: I have no idea what Baikal (Sabre/DAV) does not like about the reports, they look fine to me. Since it works fine on my end with the same baikal version: Are you using the Baikal distribution from https://github.com/sabre-io/Baikal/releases/tag/0.7.1, or are you using a different source such as a linux distribution package? My best guess at the moment is that this may be a server-side issue that has been fixed in the meantime.

Point 2: The fallback to individual GET requests does not work in the carddavclient library as I intended. I'll look into that. But then sync would be working, but slow compared to the fast paths.

[mstilkerich]

If you run composer update --no-dev, you should get the fix for point 2 and have at least a working sync. (It should install mstilkerich/carddavclient v0.1.4)

[mstilkerich]

I upgraded my dev VM to Ubuntu 20.04 which has php 7.4 and probably a newer curl and now have the same issue. So I can look into it myself.

[mstilkerich]

Seems to be an interoperability issue between curl and sabre/dav, already reported several times:
sabre-io/dav#1244 sabre-io/dav#932

And we also have an analysis already in the issues of this project: #188

So it seems to be an interoperability issue between CURL and Sabre/DAV, and it doesn't look like either side is willing to do something about it.

Option 1: Use Basic authentication in Baikal (in combination with HTTPS)
Option 2: In the Sabre/DAV installation used by Baikal, set $allowUnauthenticatedAccess = false in sabre/dav/lib/DAVACL/Plugin.php (or file an issue with Baikal to allow this by configuration. Nextcloud also uses Sabre/DAV, but changes this setting to false.).

I will see if I can force curl to send the body without slowing down all other servers by repeated probe requests (CURLAUTH_ANY would work, but do just that).

[mstilkerich]

Closing this as a duplicate of #188.

[mstilkerich]

carddavclient 1.0.0 now includes a workaround to enable usage of the fast sync methods with sabre dav.