[full-ci] - use KQL as default search query language

[fschade]

Description

Use KQL as the default oCIS search query language.

Some examples of a valid KQL query are:

• Tag: tag:golden tag:"silver"
• Filename: name:file.txt name:"file.docx"
• Content: content:ahab content:"captain aha*"

Conjunctive normal form queries:

• Boolean: tag:golden AND tag:"silver, tag:golden OR tag:"silver, tag:golden NOT tag:"silver
• Group: (tag:book content:ahab*), tag:(book pdf)

Complex queries:

• (name:"moby di*" OR tag:bestseller) AND tag:book NOT tag:read

Related Issue

• [full-ci] - use KQL as default search query language #7212
• [ocis] Use KQL for existing search syntax #7042
• [ocis]enable KQL in ocis #7179
• [search] No response from server while searching inside a folder containing space in name #7114
• [full-ci] - use KQL as default search query language web#9653
• enhancement: Keyword Query Language (KQL) search syntax support #7043
• [web] change bleve query language to KQL web#9636
• Search: scope select has wrong initial state web#9646

Motivation and Context

Defining a own query language is error prone and leads to a lot of documentation, the oCIS team decided in that ADR to use KQL as our standard. A clear set of rules is set by MS and the dictionary fits well in our usecase.

How Has This Been Tested?

• unit tests added, have been adjusted
• acceptance tests have been adjusted

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• ~~Documentation ticket raised: ~~

[fschade]

i found one missing part, i'm on it:

ocis/services/search/pkg/search/service.go

Line 421 in cd7dd56

Query: "+ID:" + storagespace.FormatResourceID(*info.Id) + ` +Mtime:>="` + utils.TSToTime(info.Mtime).Format(time.RFC3339Nano) + `"`,

[ScharfViktor]

ops, found one deficiency.
You have switched web to use space dav path, but tests still use only the new DAV path. we need to change it for all tests from apiSpaceSearch.feature like:

Scenario Outline: user can find data from the project space
    Given using <dav-path-version> DAV path
    When user "Alice" searches for "*fol*" using the WebDAV API
    Then the HTTP status code should be "207"
    And the search result should contain "4" entries
    And the search result of user "Alice" should contain these entries:
      | /folderMain                                           |
      | /folderMain/SubFolder1                                |
      | /folderMain/SubFolder1/subFOLDER2                     |
      | /folderMain/SubFolder1/subFOLDER2/insideTheFolder.txt |
    Examples:
      | dav-path-version |
      | old              |
      | new              |
    @skipOnRevaMaster
    Examples:
      | dav-path-version |
      | spaces           |

[fschade]

ops, found one deficiency.
You have switched web to use space dav path, but tests still use only the new DAV path. we need to change it for all tests from apiSpaceSearch.feature like:

@ScharfViktor, not exactly sure what you mean, but if you mean that web uses the 'space/*' endpoint for report requests, ...

it's been like that on web master since 5 weeks owncloud/web#9500

I think that also has something to do with the fact that we were on the 7.1 web release in ocis for quite a long time... sorry :(

[fschade]

would be nice to have... but it's also not highest prio since we have fallbacks in ocis....

ocis/services/webdav/pkg/service/v0/service.go

Line 109 in 3bf5e5e

r.MethodFunc("REPORT", "/remote.php/dav/spaces*", svc.Search)

the webdav endpoints won't go away, but we shift our focus to a graph api implementation from now on :)

[ScharfViktor]

@ScharfViktor, not exactly sure what you mean, but if you mean that web uses the 'space/*' endpoint for report requests, ...

yes, web uses the 'space/*' but api tests use only new DAV Path. I changed it here 109b70e and I'll create PR
after merging your

[fschade]

thanks 😗

[fschade]

@2403905, @aduffeck i had to add dateTime support to the kql <-> bleve parser, compiler, and normalizer.

can you please have a look if it's ok for you, specially because of:

ocis/services/search/pkg/search/service.go

Line 421 in cd7dd56

Query: "+ID:" + storagespace.FormatResourceID(*info.Id) + ` +Mtime:>="` + utils.TSToTime(info.Mtime).Format(time.RFC3339Nano) + `"`,

we should be ok now, i need another web bump because of:

owncloud/web#9653

can you help @janackermann, @JammingBen

[aduffeck]

lgtm from reading the code but I haven't had a chance to test it yet.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

85.9% Coverage
0.0% Duplication

[kobergj]

Awesome 👍

[2403905]

We also have issue with the implicit logical operators
(tag:klass10) NOT tag:physik will be transeted to tag:klass10 AND NOT tag:physik
But
tag:klass10 NOT tag:physik will be transeted to tag:klass10 OR NOT tag:physik

[fschade]

@2403905 thats fine! the normalization could be improved, yes, but the current clients are only sending simple queries (i hope this changes soon ;) ), lets close (merge) this here, improve our tests as a first step and then add the implementations step by step....

we first need to know (find out), what is really needed.