Implement webdav SEARCH #3360

icewind1991 · 2017-02-02T17:28:36Z

Still WIP, but submitting so the clients can start with it, functionality wise it should be ready for clients to start using the API

requires add icewind/searchdav 3rdparty#34
needs proper test coverage
Adds a mechanism to describe more complex search queries
Extend the cache to support doing these searches
Expose the new search mechanism in \OCP\Files\Folder
Implement rfc5323 SEARCH to expose this to the world
fixes Search API #676

Clients can do a search by firing a SEARCH request at nextcloud/remote.php/dav with a search body as described by the rfc

For an example of a somewhat complex query that is possible with this api see this example request

cc @mario

mention-bot · 2017-02-02T17:28:39Z

@icewind1991, thanks for your PR! By analyzing the history of the files in this pull request, we identified @MorrisJobke, @butonic and @nickvergessen to be potential reviewers.

icewind1991 · 2017-02-08T16:56:13Z

Fixes #3253

MorrisJobke

Doesn't work:

$ curl -X SEARCH http://192.168.99.100/server/remote.php/dav -u admin:admin    
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotImplemented</s:exception>
  <s:message>There was no plugin in the system that was willing to handle this SEARCH method.</s:message>
</d:error>

icewind1991 · 2017-02-15T13:10:03Z

@MorrisJobke you need to send a request body

curl -u test:test -X SEARCH  https://localcloud.icewind.me/remote.php/dav -H "content-Type: text/xml" --data '<?xml version="1.0"?>
 <d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
 <d:basicsearch>
     <d:select>
         <d:prop>
             <oc:fileid/>
             <d:getcontenttype/>
             <d:getetag/>
             <oc:size/>
         </d:prop>
     </d:select>
     <d:from>
         <d:scope>
             <d:href>/files/test</d:href>
             <d:depth>infinity</d:depth>
         </d:scope>
     </d:from>
     <d:where>
         <d:like>
             <d:prop>
                 <d:getcontenttype/>
             </d:prop>
             <d:literal>text/%</d:literal>
         </d:like>
     </d:where>
     <d:orderby/>
</d:basicsearch>
</d:searchrequest>'

MorrisJobke · 2017-02-15T13:16:53Z

I send it but it seems I forgot the text/xml header - let me retry then

LukasReschke · 2017-02-15T13:21:55Z

apps/dav/lib/Files/FileSearchBackend.php

+		/** @var Folder $folder $results */
+		$results = $folder->search($query);
+
+		var_dump(array_map(function (Node $node) {


expected? 😉

Leaving debug code in production saves me from having to create a debug patch in the future 😄

MorrisJobke · 2017-02-15T16:09:27Z

If I create a folder files/test I only get this as answer:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>InvalidArgumentException</s:exception>
  <s:message>Search is only supported on directories</s:message>
</d:error>

icewind1991 · 2017-02-15T16:18:47Z

@MorrisJobke can you paste the full request you made?

MorrisJobke · 2017-02-15T16:37:38Z

$ curl -X SEARCH https://example.org/remote.php/dav -u admin:admin -i -H "content-Type: text/xml" --data '<?xml version="1.0"?> 
 <d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
 <d:basicsearch>
     <d:select>
         <d:prop>
             <oc:fileid/>
             <d:getcontenttype/>
             <d:getetag/>
             <oc:size/>
         </d:prop>
     </d:select>
     <d:from>
         <d:scope>
             <d:href>/files/test</d:href>
             <d:depth>infinity</d:depth>
         </d:scope>
     </d:from>
     <d:where>
         <d:like>
             <d:prop>
                 <d:getcontenttype/>
             </d:prop>
             <d:literal>text/%</d:literal>
         </d:like>
     </d:where>
     <d:orderby/>
</d:basicsearch>
</d:searchrequest>'

HTTP/1.1 404 Not Found
Date: Wed, 15 Feb 2017 16:36:52 GMT
...
Content-Length: 226
Content-Type: application/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>Principal with name test not found</s:message>
</d:error>

MorrisJobke · 2017-02-15T16:40:58Z

icewind1991 · 2017-02-17T21:59:32Z

@MorrisJobke the scope paths are relative to the dav root (remote.php/dav), so settings the scope to files/test searches in the root folder of the test user

MorrisJobke · 2017-02-20T19:04:06Z

@MorrisJobke the scope paths are relative to the dav root (remote.php/dav), so settings the scope to files/test searches in the root folder of the test user

Tested and works 👍

cc @marinofaggiana I tested it against your test setup and it works 😃

codecov-io · 2017-02-21T18:39:19Z

Codecov Report

Merging #3360 into master will increase coverage by 0.54%.
The diff coverage is 68.96%.

@@             Coverage Diff             @@
##             master   #3360      +/-   ##
===========================================
+ Coverage     54.15%   54.7%   +0.54%     
- Complexity    21023   21630     +607     
===========================================
  Files          1306    1316      +10     
  Lines         80264   82610    +2346     
  Branches       1250    1300      +50     
===========================================
+ Hits          43466   45190    +1724     
- Misses        36798   37420     +622

Impacted Files	Coverage Δ	Complexity Δ
lib/private/Files/Search/SearchOrder.php	`0% <ø> (ø)`	`3 <3> (?)`
apps/dav/lib/Server.php	`45.31% <ø> (-2.63%)`	`12 <ø> (ø)`
lib/private/Lockdown/Filesystem/NullCache.php	`90.9% <ø> (-3.44%)`	`22 <1> (+1)`
lib/private/Files/Cache/FailedCache.php	`0% <ø> (ø)`	`26 <1> (+1)`	✅
lib/private/Files/Search/SearchComparison.php	`100% <100%> (ø)`	`4 <4> (?)`
lib/private/Files/Cache/Wrapper/CacheJail.php	`85.48% <100%> (+0.11%)`	`55 <1> (+18)`	✅
lib/private/Files/Cache/Wrapper/CacheWrapper.php	`78.08% <100%> (+0.93%)`	`32 <1> (+1)`	✅
lib/private/Files/Node/Folder.php	`88.05% <50%> (-0.83%)`	`49 <ø> (+1)`
lib/private/Files/Search/SearchBinaryOperator.php	`57.14% <57.14%> (ø)`	`3 <3> (?)`
lib/private/Files/Cache/QuerySearchHelper.php	`67.12% <67.12%> (ø)`	`23 <23> (?)`
... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd49eb3...7c7abf7. Read the comment docs.

MorrisJobke · 2017-02-22T01:04:23Z

For this body:

<?xml version="1.0"?>
<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:basicsearch>
    <d:select>
      <d:prop>
        <d:displayname/>
      </d:prop>
    </d:select>
    <d:from>
      <d:scope>
        <d:href>/files/admin</d:href>
        <d:depth>infinity</d:depth>
      </d:scope>
    </d:from>
    <d:where>
      <d:like>
        <d:prop>
          <d:displayname/>
        </d:prop>
        <d:literal>Documents</d:literal>
      </d:like>
    </d:where>
  </d:basicsearch>
</d:searchrequest>

I get following response:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
  <s:message>TypeError: Argument 4 passed to OC\Files\Search\SearchQuery::__construct() must be of the type array, null given, called in /var/www/nextcloud/apps/dav/lib/Files/FileSearchBackend.php on line 177</s:message>
</d:error>

LukasReschke · 2017-02-22T09:12:03Z

<s:message>TypeError: Argument 4 passed to OC\Files\Search\SearchQuery::__construct() must be of the type array, null given, called in /var/www/nextcloud/apps/dav/lib/Files/FileSearchBackend.php on line 177</s:message>

General note: Can we avoid leaking exception messages via DAV? They can sometimes contain sensitive content :)

icewind1991 · 2017-02-22T17:25:17Z

@MorrisJobke can you retry with the updated 3rdparty

MorrisJobke · 2017-02-24T06:13:40Z

@MorrisJobke can you retry with the updated 3rdparty

I now get:

HTTP/1.1 207 Multi-Status

<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:cs="http://calendarserver.org/ns/" xmlns:card="urn:ietf:params:xml:ns:carddav" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns"/>

😕

icewind1991 · 2017-02-24T14:13:23Z

@MorrisJobke what is the request you're trying?

MorrisJobke · 2017-02-28T05:25:05Z

@MorrisJobke what is the request you're trying?

#3360 (comment)

mario · 2017-02-28T11:38:58Z

@icewind1991 when doing OPTIONS, search doesn't seem to be among supported methods?

icewind1991 · 2017-02-28T12:18:30Z

Works for me:

curl -I -X OPTIONS  -u test:test https://localcloud.icewind.me/remote.php/dav
...
Allow: OPTIONS, GET, HEAD, DELETE, PROPFIND, PUT, PROPPATCH, COPY, MOVE, REPORT, SEARCH

mario · 2017-02-28T12:25:17Z

Ok will check once again.

…

On Tue, 28 Feb 2017 at 13:18, Robin Appelman ***@***.***> wrote: Works for me: curl -I -X OPTIONS -u test:test https://localcloud.icewind.me/remote.php/dav ... Allow: OPTIONS, GET, HEAD, DELETE, PROPFIND, PUT, PROPPATCH, COPY, MOVE, REPORT, SEARCH — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3360 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAWsubNeTyFPZjT7PiHSMFwDzucPUUKks5rhBCYgaJpZM4L1ZwO> .

mario · 2017-02-28T13:01:59Z

@icewind1991 would something like this work with your implementation, since it doesn't look like the current libs supports that draft entirely:

<d:searchrequest xmlns:d="DAV:" dcr:="http://www.day.com/jcr/webdav/1.0" >
dcr:xpath//sv:node[@sv:name='myapp:paragraph'][1]</dcr:xpath>
</d:searchrequest>

cc @AndyScherzinger @tobiasKaminsky

Otherwise we might look into how we can either extend it use or use (possibly?) another lib just for this particular use case if such a thing exists.

Signed-off-by: Robin Appelman <robin@icewind.nl>

marinofaggiana · 2017-03-06T14:43:53Z

ping @icewind1991

icewind1991 · 2017-03-06T14:43:56Z

@marinofaggiana please describe the error you're getting instead of having us guess

marinofaggiana · 2017-03-06T14:46:53Z

curl -u admin:XXXXXXXX -X SEARCH https://ios-demo.weasel.rocks/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/admin</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:like><d:prop><d:displayname/></d:prop><d:literal>Nextcloud%</d:literal></d:like></d:where><d:orderby/></d:basicsearch></d:searchrequest>'

marinofaggiana · 2017-03-06T14:49:44Z

@icewind1991 this is the response :

<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotImplemented</s:exception>
<s:message>There was no plugin in the system that was willing to handle this SEARCH method.</s:message>
</d:error>

marinofaggiana · 2017-03-07T10:44:22Z

@icewind1991 @MorrisJobke please info for this issue.

mario · 2017-03-07T14:32:49Z

So search basically works, so please +1 all of you.

After that, I just need it extended to support oc:favorite in a query because filter-files won't work for Android atm.

marinofaggiana · 2017-03-07T14:47:01Z

@mario
"<oc:filter-files xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns\" xmlns:nc="http://nextcloud.org/ns\">oc:filter-rulesoc:favorite1</oc:favorite></oc:filter-rules><d:prop>"

mario · 2017-03-07T14:50:21Z

@marinofaggiana using that our Android lib doesn't want to parse the propstat so I don't get the properties back.

marinofaggiana · 2017-03-07T14:52:37Z

Modify the lib ?

mario · 2017-03-07T15:09:03Z

Not something I feel comfortable doing under these kind of deadlines.

MorrisJobke · 2017-03-08T00:03:24Z

Sorry - I screwed up the instance. It should work now. :( @marinofaggiana @icewind1991

barrydegraaff · 2018-01-15T13:28:14Z

@mario good work. I have been playing around with WebDAV SEARCH on nextcloud 12.0.4 and integrate it in Zimbra Nextcloud Client (https://github.com/Zimbra-Community/owncloud-zimlet)

Mostly I can make it work, but I have issues searching for files with more than 2 search words.
So I would like to search for all files that match: word1, word2 and word3. So far I can only make Nextcloud look for word1 and word2. The 3rd word is ignored. Where can I find good documentation on webdav search?

   curl -u user:password -X SEARCH https://nextcloud.example.com/nextcloud/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontentlength/><d:getlastmodified/><d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/user</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where><d:orderby><d:order><d:prop><d:displayname/></d:prop><d:ascending/></d:order><d:order><d:prop><d:getlastmodified/></d:prop><d:ascending/></d:order></d:orderby></d:basicsearch></d:searchrequest>'


   curl -u user:password -X SEARCH https://nextcloud.domain.com/nextcloud/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontentlength/><d:getlastmodified/><d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/user</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like></d:and></d:where><d:orderby><d:order><d:prop><d:displayname/></d:prop><d:ascending/></d:order><d:order><d:prop><d:getlastmodified/></d:prop><d:ascending/></d:order></d:orderby></d:basicsearch></d:searchrequest>'

So here only the first 2 words are matched, the 3rd is ignored... is there a limit of 2 in AND?

     <d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where>

barrydegraaff · 2018-01-15T13:37:47Z

https://www.ietf.org/rfc/rfc5323.txt
5.7. Boolean Operators: DAV:and, DAV:or, and DAV:not

The DAV:and operator performs a logical AND operation on the
expressions it contains.

Can anyone help me to do a WebDAV Search to nextcloud with more than 2 words to look for in displayname, I think the XML example, is according to the spec, but maybe I misread.

  <d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where>

icewind1991 · 2018-01-15T13:40:20Z

How I understood the spec (and implemented it) is that DAV:and and DAV:or only take 2 arguments,
you can nest multiple and/or statements to get your desired effect

barrydegraaff · 2018-01-15T13:50:57Z

@icewind1991 thanks, that makes it a bit harder to implement on the client side. It would be nice if the rfc said The operator takes two arguments. instead of The DAV:and operator performs a logical AND operation on the expressions it contains. I would not read 2 in that.

barrydegraaff · 2018-01-15T14:05:48Z

@icewind1991 I am thinking about asking julian reschke (his name is on the rfc) and see what he says.

But should the server return a 501 Not Implemented error if I do something that is outside the webdav spec? Aka an AND with 3 criteria = 501. Now it parses only the first 2.

reschke · 2018-01-15T14:34:26Z

I agree that RFC 5323 could be clearer, but AFAIR, we would have said "two" if we had intended to restrict it to two operands.

icewind1991 · 2018-01-15T20:04:02Z

I will see if I can extend the implementation to allow for arbitrary number of arguments

barrydegraaff · 2018-01-16T08:29:42Z

@icewind1991 thanks!

rullzer · 2018-01-16T11:55:21Z

I just noticed this as well.

Since and is associative you can just do

a AND b AND c AND d

and transform it into

((a AND b) AND c) AND d)

icewind1991 · 2018-01-16T12:25:16Z

Fix is here

rullzer · 2018-02-25T18:39:52Z

@pinkylabs please open a new issue.

icewind1991 mentioned this pull request Feb 2, 2017

add icewind/searchdav nextcloud/3rdparty#34

Merged

MorrisJobke mentioned this pull request Feb 2, 2017

Search API #676

Closed

MorrisJobke requested changes Feb 15, 2017

View reviewed changes

LukasReschke reviewed Feb 15, 2017

View reviewed changes

icewind1991 force-pushed the dav-search branch from a7535fc to 2d4cb4d Compare February 15, 2017 13:28

mario mentioned this pull request Feb 28, 2017

File search nextcloud/android#558

Closed

icewind1991 added this to the Nextcloud 12.0 milestone Feb 28, 2017

add icewind/searchdav

706131b

Signed-off-by: Robin Appelman <robin@icewind.nl>

icewind1991 force-pushed the dav-search branch from caab1f9 to 56f84d6 Compare February 28, 2017 13:24

rullzer approved these changes Mar 8, 2017

View reviewed changes

icewind1991 merged commit 2a8e922 into master Mar 8, 2017

icewind1991 deleted the dav-search branch March 8, 2017 12:09

MorrisJobke mentioned this pull request Jun 16, 2017

Document WebDAV search nextcloud/documentation#504

Closed

icewind1991 mentioned this pull request Jan 16, 2018

Support arbitrary number of arguments for d:or and d:and in search queries #7882

Merged

pinkylabs mentioned this pull request Feb 25, 2018

WebDAV SEARCH request requires 'where' to be present, which is not actually required by rfc #8531

Open

Implement webdav SEARCH #3360

Implement webdav SEARCH #3360

Conversation

icewind1991 commented Feb 2, 2017 • edited by MorrisJobke Loading

mention-bot commented Feb 2, 2017

icewind1991 commented Feb 8, 2017

MorrisJobke left a comment

Choose a reason for hiding this comment

icewind1991 commented Feb 15, 2017 • edited Loading

MorrisJobke commented Feb 15, 2017

LukasReschke Feb 15, 2017

Choose a reason for hiding this comment

icewind1991 Feb 15, 2017

Choose a reason for hiding this comment

MorrisJobke commented Feb 15, 2017

icewind1991 commented Feb 15, 2017

MorrisJobke commented Feb 15, 2017

MorrisJobke commented Feb 15, 2017

icewind1991 commented Feb 17, 2017

MorrisJobke commented Feb 20, 2017

codecov-io commented Feb 21, 2017

Codecov Report

MorrisJobke commented Feb 22, 2017 • edited Loading

LukasReschke commented Feb 22, 2017 • edited Loading

icewind1991 commented Feb 22, 2017

MorrisJobke commented Feb 24, 2017

icewind1991 commented Feb 24, 2017

MorrisJobke commented Feb 28, 2017

mario commented Feb 28, 2017

icewind1991 commented Feb 28, 2017

mario commented Feb 28, 2017 via email

mario commented Feb 28, 2017

marinofaggiana commented Mar 6, 2017

icewind1991 commented Mar 6, 2017

marinofaggiana commented Mar 6, 2017

marinofaggiana commented Mar 6, 2017

marinofaggiana commented Mar 7, 2017

mario commented Mar 7, 2017

marinofaggiana commented Mar 7, 2017 • edited Loading

mario commented Mar 7, 2017

marinofaggiana commented Mar 7, 2017

mario commented Mar 7, 2017

MorrisJobke commented Mar 8, 2017

barrydegraaff commented Jan 15, 2018 • edited Loading

barrydegraaff commented Jan 15, 2018

icewind1991 commented Jan 15, 2018

barrydegraaff commented Jan 15, 2018

barrydegraaff commented Jan 15, 2018

reschke commented Jan 15, 2018

icewind1991 commented Jan 15, 2018

barrydegraaff commented Jan 16, 2018 via email

rullzer commented Jan 16, 2018

icewind1991 commented Jan 16, 2018

rullzer commented Feb 25, 2018

icewind1991 commented Feb 2, 2017 •

edited by MorrisJobke

Loading

icewind1991 commented Feb 15, 2017 •

edited

Loading

MorrisJobke commented Feb 22, 2017 •

edited

Loading

LukasReschke commented Feb 22, 2017 •

edited

Loading

marinofaggiana commented Mar 7, 2017 •

edited

Loading

barrydegraaff commented Jan 15, 2018 •

edited

Loading