Sabre Update to 2.1 #14151

PVince81 · 2015-02-12T11:34:00Z

Fixes Update SabreDAV to 2.0 #12876
Requires: Update to Sabre 2.1 owncloud-archive/3rdparty#158

TODOs

fix connector classes
fix auth
...

Regression testing

publicwebdav / s2s
webdav operations
sync client (chunking, parallel upload, etc)
caldav: follow up in LDAP users: UUID instead of displayName on Calendar app calendar#722
carddav: follow up in https://github.com/owncloud/contacts/pull/826

DeepDiver1975 · 2015-02-12T12:06:09Z

lib/private/connector/sabre/server.php

@@ -1,4 +1,6 @@
 <?php
+namespace OC\Connector\Sabre;


I'd be fine to kill the connector from the namespace and folder structure

Next time maybe, there's already enough to do to makes this work 😉

I'll namespacize the classes I touch though.

PVince81 · 2015-02-12T12:23:55Z

use another approach for OC_xxx_Request class (getRawServerValue())
~~retest "Principal" stuff~~
double check "$propertyMap" in TagList
check anythng VObject related

PVince81 · 2015-02-12T13:03:34Z

test: quota

PVince81 · 2015-02-12T16:08:52Z

test: oc_properties
test: tagsplugin

PVince81 · 2015-02-12T16:18:28Z

test: range requests

It looks like the methods "httpGet", etc have to be moved to a plugin now.

PVince81 · 2015-02-12T16:35:11Z

find a way to override depth to keep the "infinity" depth feature (which is why we seem to have patched server.php: https://github.com/owncloud/core/blob/master/lib/private/connector/sabre/server.php#L17 https://github.com/owncloud/core/blob/master/lib/private/connector/sabre/server.php#L102

PVince81 · 2015-02-12T16:35:42Z

Oh, that is nice:

        if (!$this->server->enablePropfindDepthInfinity && $depth != 0) $depth = 1;

PVince81 · 2015-02-12T16:49:50Z

double check all returned properties (because I removed our overridden methods and the Sabre Server code changed a lot)

PVince81 · 2015-02-12T18:06:50Z

Looks like the "beforeGetProperties" approach isn't valid any more. Need to switch to another technique, maybe listenning to the "propFind" even on nodes.

move tags plugin to use "propFind" event instead
double check files plugin (need to return file id, etc) due to arch changes

PVince81 · 2015-02-12T18:08:56Z

adapt custom props code to use "propFind" event as well

evert · 2015-02-12T18:26:58Z

lib/private/connector/sabre/filesplugin.php

-		$this->server->subscribeEvent('beforeGetProperties', array($this, 'beforeGetProperties'));
-		$this->server->subscribeEvent('afterBind', array($this, 'sendFileIdHeader'));
-		$this->server->subscribeEvent('afterWriteContent', array($this, 'sendFileIdHeader'));
+		$this->server->on('beforeGetProperties', array($this, 'beforeGetProperties'));


Note that beforeGetProperties has been removed as of version 2.0. There's a new event called propFind that does what you need, and it's documented here:

http://sabre.io/dav/writing-plugins/

Thanks for the hint, already found it and implementing it now.

How about "beforeGetPropertiesForPath" ? That one was convenient for prefeteching data for multiple nodes in a go. Is that still possible ?

Yes, the idea here was that we simply still emit a propFind event. This event has an argument that's an instance of Sabre\DAV\PropFind. That object will have a depth property set to 1, which means that you'll need to prefetch the data. At least, that was the intention. I didn't test this myself!

PVince81 · 2015-02-12T18:37:52Z

evaluate whether our "Tree" implementation is still needed: seems we still need it. To remove it, need deeper analysis of the logic + perf analysis, and I don't want to spend too much time on this now.

evert · 2015-02-12T18:38:17Z

lib/private/vobject/compoundproperty.php

@@ -26,7 +26,7 @@
 * This class overrides \Sabre\VObject\Property::serialize() to not
 * double escape commas and semi-colons in compound properties.
 */
-class CompoundProperty extends \Sabre\VObject\Property\Compound {
+class CompoundProperty extends \Sabre\VObject\Property\Text {


This existed to work around issues with vobject 2.1. These issues should no longer exist, so should be removed.

PVince81 · 2015-02-12T18:43:13Z

get rid of VObject custom types (CompoundObject and the other one)
find replacement for "beforeGetPropertiesForPath" to pre-cache tags (as we can read multiple at the same time from the DB)

evert · 2015-02-12T19:08:33Z

I'm pretty sure you can integrate scrutinizer properly with github these days, so the issue comments are no longer needed. They're a bit useless!

evert · 2015-02-12T19:11:54Z

Same with jenkins! So noisy!

DeepDiver1975 · 2015-02-12T19:22:11Z

The jenkins github integration does not allow multiple commit statuses. Maybe one day .....

evert · 2015-02-12T20:16:26Z

Very awesome to see that this is a 'negative diff' so far ;) More lines removed than added.

PVince81 · 2015-02-13T12:17:05Z

put etag through PROPPATCH (formerly done with ugly oc_properties logic)

PVince81 · 2015-02-13T12:17:23Z

put mtime through PROPPATCH (formerly done with ugly oc_properties logic)

PVince81 · 2015-02-13T12:25:12Z

@evert if I implement the custom properties logic in a plugin (a backend of PropertyStorage), should I remove "implement IProperties" from the Node then ?

PVince81 · 2015-02-13T16:20:13Z

~~abuse allprops to also return "oc:id", "oc:permissions" and "oc:size" (worked on master)~~ raise ticket for clients to always explicitly request these attrs (they'll need anyway once they start using oc:favorite, tags, etc)

evert · 2015-02-13T16:25:47Z

@evert if I implement the custom properties logic in a plugin (a backend of PropertyStorage), should I remove "implement IProperties" from the Node then ?

Keeping it is optional. The new property system is designed so multiple separate plugins can 'opt in' to handling storing or retrieval of certain properties. If you no longer have a need for IProperties in nodes, you can remove it.

PVince81 · 2015-02-13T16:38:39Z

@DeepDiver1975 I will not try and refactor the chunking into plugins right now to reduce the risk of regressions. I think we anyway need to rethink the way we do chunking, maybe send headers instead of using magic file names that need to be parsed. (I tried rewriting the URL in a beforeMethod but it messes up other things, so no easy fix here)

DeepDiver1975 · 2015-02-23T22:27:10Z

Not reproducible while debugging - sounds like a timing issue - I hate it when that 💩 happens

DeepDiver1975 · 2015-02-23T22:28:56Z

I'll continue tomorrow ....

@evert Many thanks for your support on this! Much appreciated!

DeepDiver1975 · 2015-02-25T09:14:51Z

well - the errors as of above and no longer reproducible:

deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ litmus -k https://localhost/oc/core/remote.php/webdav/ admin admin
-> running `copymove':
 0. init.................. pass
 1. begin................. pass
 2. copy_init............. pass
 3. copy_simple........... pass
 4. copy_overwrite........ pass
 5. copy_nodestcoll....... pass
 6. copy_cleanup.......... pass
 7. copy_coll............. pass
 8. copy_shallow.......... pass
 9. move.................. pass
10. move_coll............. pass
11. move_cleanup.......... pass
12. finish................ pass
<- summary for `copymove': of 13 tests run: 13 passed, 0 failed. 100.0%
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ gedit debug.log 
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ gedit debug.log 
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ rm debug.log 
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ litmus -k https://localhost/oc/core/remote.php/webdav/ admin admin
-> running `copymove':
 0. init.................. pass
 1. begin................. pass
 2. copy_init............. pass
 3. copy_simple........... pass
 4. copy_overwrite........ pass
 5. copy_nodestcoll....... pass
 6. copy_cleanup.......... pass
 7. copy_coll............. pass
 8. copy_shallow.......... pass
 9. move.................. pass
10. move_coll............. pass
11. move_cleanup.......... pass
12. finish................ pass
<- summary for `copymove': of 13 tests run: 13 passed, 0 failed. 100.0%
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ rm debug.log 
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ litmus -k https://localhost/oc/core/remote.php/webdav/ admin admin
-> running `copymove':
 0. init.................. pass
 1. begin................. pass
 2. copy_init............. pass
 3. copy_simple........... pass
 4. copy_overwrite........ pass
 5. copy_nodestcoll....... pass
 6. copy_cleanup.......... pass
 7. copy_coll............. pass
 8. copy_shallow.......... pass
 9. move.................. pass
10. move_coll............. pass
11. move_cleanup.......... pass
12. finish................ pass
<- summary for `copymove': of 13 tests run: 13 passed, 0 failed. 100.0%
deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$

DeepDiver1975 · 2015-02-25T09:22:02Z

All unit tests are successfully executed now as well

deepdiver@alien:~/Development/ownCloud/core [update-sabre2.1]$ litmus -k https://localhost/oc/core/remote.php/webdav/ admin admin
-> running `basic':
 0. init.................. pass
 1. begin................. pass
 2. options............... WARNING: server does not claim Class 2 compliance
    ...................... pass (with 1 warning)
 3. put_get............... pass
 4. put_get_utf8_segment.. pass
 5. put_no_parent......... pass
 6. mkcol_over_plain...... pass
 7. delete................ pass
 8. delete_null........... pass
 9. delete_fragment....... pass
10. mkcol................. pass
11. mkcol_again........... pass
12. delete_coll........... pass
13. mkcol_no_parent....... pass
14. mkcol_with_body....... pass
15. finish................ pass
<- summary for `basic': of 16 tests run: 16 passed, 0 failed. 100.0%
-> 1 warning was issued.
-> running `copymove':
 0. init.................. pass
 1. begin................. pass
 2. copy_init............. pass
 3. copy_simple........... pass
 4. copy_overwrite........ pass
 5. copy_nodestcoll....... pass
 6. copy_cleanup.......... pass
 7. copy_coll............. pass
 8. copy_shallow.......... pass
 9. move.................. pass
10. move_coll............. pass
11. move_cleanup.......... pass
12. finish................ pass
<- summary for `copymove': of 13 tests run: 13 passed, 0 failed. 100.0%
-> running `props':
 0. init.................. pass
 1. begin................. pass
 2. propfind_invalid...... pass
 3. propfind_invalid2..... pass
 4. propfind_d0........... pass
 5. propinit.............. pass
 6. propset............... pass
 7. propget............... pass
 8. propextended.......... pass
 9. propmove.............. pass
10. propget............... pass
11. propdeletes........... pass
12. propget............... pass
13. propreplace........... pass
14. propget............... pass
15. propnullns............ pass
16. propget............... pass
17. prophighunicode....... pass
18. propget............... pass
19. propremoveset......... pass
20. propget............... pass
21. propsetremove......... pass
22. propget............... pass
23. propvalnspace......... pass
24. propwformed........... pass
25. propinit.............. pass
26. propmanyns............ pass
27. propget............... pass
28. propcleanup........... pass
29. finish................ pass
<- summary for `props': of 30 tests run: 30 passed, 0 failed. 100.0%
-> running `locks':
 0. init.................. pass
 1. begin................. pass
 2. options............... WARNING: server does not claim Class 2 compliance
    ...................... pass (with 1 warning)
 3. precond............... SKIPPED (locking tests skipped,
server does not claim Class 2 compliance)
-> 1 test was skipped.
<- summary for `locks': of 3 tests run: 3 passed, 0 failed. 100.0%
-> 1 warning was issued.
-> running `http':
 0. init.................. pass
 1. begin................. pass
 2. expect100............. SKIPPED (skipping for SSL server)
 3. finish................ pass
-> 1 test was skipped.
<- summary for `http': of 3 tests run: 3 passed, 0 failed. 100.0%

DeepDiver1975 · 2015-02-25T09:22:42Z

@PVince81 from my pov this is ready to go - anything missing?

Reasoning: - a WebDAV server is not required to implement locking support - WebDAV Locking is know to break the sync algorithm - the current lock implementation is known to be broken (locks are not moved if a file is moved, locks on shared files don't work)

PVince81 · 2015-02-25T09:32:10Z

I hereby agree with your commit that removes the locks 👍

PVince81 · 2015-02-25T09:33:39Z

Jenkins PR: #14483

scrutinizer-notifier · 2015-02-25T09:58:57Z

The inspection completed: 25 new issues, 74 updated code elements

DeepDiver1975 · 2015-02-25T10:19:19Z

Refer to this link for build results (access rights to CI server needed):
https://ci.owncloud.org//job/pull-request-analyser-ng-simple/9932/
Test PASSed.

DeepDiver1975 · 2015-02-25T10:21:52Z

👍

Sabre Update to 2.1

evert · 2015-02-25T17:54:30Z

Hi guys:

I would not take the earlier litmus failure lightly. If you don't have a reasonable explanation why it happened in the first place, I suspect there may be some race condition and I wouldn't ignore it.
Locking support is indeed not required, but a lot of clients do want it to be there. OS X finder for example will only allow read-only access to webdav if there's no locking support.

DeepDiver1975 · 2015-02-25T20:08:38Z

Hi guys:

I would not take the earlier litmus failure lightly. If you don't have a reasonable explanation why it happened in the first place, I suspect there may be some race condition and I wouldn't ignore it.

Sure thing - but we need to go ahead as a hell of other tasks depend on this change.
Furthermore - having the code in master will give it some more test coverage and thus more reliability.

Locking support is indeed not required, but a lot of clients do want it to be there. OS X finder for example will only allow read-only access to webdav if there's no locking support.

Sure - but I prefer to lock a file the right way and not that crappy way we do today where we - in some case - pretend to lock but we are actually not locking at all.

PVince81 · 2015-02-25T20:14:57Z

Does litmus do concurrency tests ? If not I don't see how a race condition could happen.

Ideally the "new" locking (if ever) should be based on file id for better accuracy even with shared files, but even then it requires stable file ids first.

DeepDiver1975 · 2015-02-25T20:27:34Z

Does litmus do concurrency tests ?

no the tests are executed sequentially

evert · 2015-02-25T20:59:06Z

Sure - but I prefer to lock a file the right way and not that crappy way we do today where we - in some case - pretend to lock but we are actually not locking at all.

I actually think that webdav locks are not really useful, so I was considering writing an alternative locks plugin that pretends to support locking, but doesn't actually do anything and requires no storage.

Clients really ought to be using ETags and such instead of doing locking.

DeepDiver1975 · 2015-02-25T21:07:01Z

From my personal pov it highly depends on the workflow - with locking changes can be prevented before they actually did take place.
Depending on the ETag will result in different version - so what's next? Displaying a diff of versions to the user is not always possible.

evert · 2015-02-25T21:08:56Z

The problem with locks and how clients generally implement it, is that they only do the LOCK moments before they do the actual PUT (or other operation) and then do an UNLOCK right after. This behavior has no benefits over using etags and if headers.

For locks to be useful, they really need to be controlled by the user before work starts and unlocked after completion,

DeepDiver1975 · 2015-02-25T21:17:08Z

For locks to be useful, they really need to be controlled by the user before work starts and unlocked after completion,

This is indeed the workflow I have in mind - 🙊 this is how M$ Office is doing it afaik

As discussed in #14151, we missed to add this to the `files_sharing` S2S public WebDAV backend though.

As discussed in owncloud#14151, we missed to add this to the `files_sharing` S2S public WebDAV backend though.

DeepDiver1975 reviewed Feb 12, 2015
View reviewed changes

PVince81 mentioned this pull request Feb 12, 2015

WebDav: Using "allprop" on PROPFIND does not behave as expected #11803

Closed

evert reviewed Feb 12, 2015
View reviewed changes

PVince81 force-pushed the update-sabre2.1 branch from 3f6700e to c6ca07c Compare February 12, 2015 19:05

Remove locks plugin.

b3de86d

Reasoning: - a WebDAV server is not required to implement locking support - WebDAV Locking is know to break the sync algorithm - the current lock implementation is known to be broken (locks are not moved if a file is moved, locks on shared files don't work)

DeepDiver1975 force-pushed the update-sabre2.1 branch from 46b6fc6 to b3de86d Compare February 25, 2015 09:31

PVince81 mentioned this pull request Feb 25, 2015

[Jenkkins] Update sabre2.1 jenkins dance #14483

Merged

DeepDiver1975 added a commit that referenced this pull request Feb 25, 2015

Merge pull request #14151 from owncloud/update-sabre2.1

799e144

Sabre Update to 2.1

DeepDiver1975 merged commit 799e144 into master Feb 25, 2015

DeepDiver1975 deleted the update-sabre2.1 branch February 25, 2015 10:22

LukasReschke mentioned this pull request Mar 2, 2015

Do authentication using another valid WebDAV call instead of GET owncloud/client#2911

Closed

RobinMcCorkell mentioned this pull request Mar 27, 2015

Add proper namespaces for the sabre connector classes #12864

Closed

LukasReschke added a commit that referenced this pull request May 14, 2015

Show a empty response for GET on non-files instead of the Browser Plugin

e0a136d

As discussed in #14151, we missed to add this to the `files_sharing` S2S public WebDAV backend though.

LukasReschke mentioned this pull request May 14, 2015

Show a empty response for GET on non-files instead of the Browser Plugin #16354

Merged

mmattel pushed a commit to mmattel/core that referenced this pull request May 22, 2015

Show a empty response for GET on non-files instead of the Browser Plugin

aae4f9d

As discussed in owncloud#14151, we missed to add this to the `files_sharing` S2S public WebDAV backend though.

lock bot locked as resolved and limited conversation to collaborators Aug 13, 2019

Sabre Update to 2.1 #14151

Sabre Update to 2.1 #14151

Conversation

PVince81 commented Feb 12, 2015

TODOs

Regression testing

DeepDiver1975 Feb 12, 2015

Choose a reason for hiding this comment

PVince81 Feb 12, 2015

Choose a reason for hiding this comment

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

PVince81 commented Feb 12, 2015

evert Feb 12, 2015

Choose a reason for hiding this comment

PVince81 Feb 12, 2015

Choose a reason for hiding this comment

evert Feb 12, 2015

Choose a reason for hiding this comment

PVince81 commented Feb 12, 2015

evert Feb 12, 2015

Choose a reason for hiding this comment

PVince81 commented Feb 12, 2015

evert commented Feb 12, 2015

evert commented Feb 12, 2015

DeepDiver1975 commented Feb 12, 2015

evert commented Feb 12, 2015

PVince81 commented Feb 13, 2015

PVince81 commented Feb 13, 2015

PVince81 commented Feb 13, 2015

PVince81 commented Feb 13, 2015

evert commented Feb 13, 2015

PVince81 commented Feb 13, 2015

DeepDiver1975 commented Feb 23, 2015

DeepDiver1975 commented Feb 23, 2015

DeepDiver1975 commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

PVince81 commented Feb 25, 2015

PVince81 commented Feb 25, 2015

scrutinizer-notifier commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

evert commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

PVince81 commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

evert commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015

evert commented Feb 25, 2015

DeepDiver1975 commented Feb 25, 2015