enhancement: Keyword Query Language (KQL) search syntax support

[fschade]

Description

Introduce support for KQL syntax for search.

The functionality consists of a kql lexer and a bleve query compiler

Supported field queries:

• Tag search tag:golden tag:"silver"
• Filename search name:file.txt name:"file.docx"
• Content search content:ahab content:"captain aha*"

Supported conjunctive normal form queries:

• Boolean operators AND, OR, NOT,
• Nesting ( ...SUB_QUERY... )

for example:

Query:

(name:"moby di*" OR tag:bestseller) AND tag:book NOT tag:read

Result:

• Resources with name: moby di* OR tag: bestseller.
• AND with tag:book.
• NOT with tag:read.

AST:

&ast.Ast{
  Nodes: []ast.Node{
    &ast.GroupNode{
      Base: &ast.Base{Loc: &ast.Location{...}},
      Nodes: []ast.Node{
        &ast.StringNode{...},
        &ast.OperatorNode{...},
        &ast.StringNode{...},
      },
    },
    &ast.OperatorNode{Base: &ast.Base{Loc: &ast.Location{...}}, Value: "AND"},
    &ast.StringNode{Base: &ast.Base{Loc: &ast.Location{...}}, Key: "tag", Value: "book"},
    &ast.OperatorNode{Base: &ast.Base{Loc: &ast.Location{...}}, Value: "NOT"},
    &ast.StringNode{Base: &ast.Base{Loc: &ast.Location{...}}, Key: "tag", Value: "read"},
  },
}

Todos:

• KQL-STRING-QUERY to AST kql syntax subset [KQL-TO-AST-LEXER]
  • name (ast.StringNode)
  • tag (ast.StringNode)
  • content (ast.StringNode)
  • free text-keywords (ast.StringNode)
  • conjunctive normal form (ast.GroupNode)
  • operators AND, OR, NOT (ast.OperatorNode)
• AST to BLEVE-STRING-QUERY compiler [AST-TO-BLEVE-COMPILER]
  • name (ast.StringNode)
  • tag (ast.StringNode)
  • content (ast.StringNode)
  • free text-keywords (ast.StringNode)
  • conjunctive normal form (ast.GroupNode)
  • operators AND, OR, NOT (ast.OperatorNode)

Related Issue

• Fixes [ocis] Use KQL for existing search syntax #7042

How Has This Been Tested?

• unit tests

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:

[2403905]

We should discuss how to handle the implicit boolean operators and NOT together. It is not explicitly described in the documentation. The rules are different for the multiple free-text expressions, multiple instances of the same property restriction, multiple instances of the different property and grouping-property.

ocis/services/search/pkg/query/kql/factory.go

Lines 156 to 176 in 46b8e43

	// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#constructing-free-text-queries-using-kql
	// If there are multiple free-text expressions without any operators in between them, the query behavior is the same as using the AND operator.
	// "John Smith" "Jane Smith"
	// This functionally is the same as using the OR Boolean operator, as follows:
	// "John Smith" AND "Jane Smith"
	//
	// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#using-multiple-property-restrictions-within-a-kql-query
	// When you use multiple instances of the same property restriction, matches are based on the union of the property restrictions in the KQL query.
	// author:"John Smith" author:"Jane Smith"
	// This functionally is the same as using the OR Boolean operator, as follows:
	// author:"John Smith" OR author:"Jane Smith"
	//
	// When you use different property restrictions, matches are based on an intersection of the property restrictions in the KQL query, as follows:
	// author:"John Smith" filetype:docx
	// This is the same as using the AND Boolean operator, as follows:
	// author:"John Smith" AND filetype:docx
	//
	// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#grouping-property-restrictions-within-a-kql-query
	// author:("John Smith" "Jane Smith")
	// This is the same as using the AND Boolean operator, as follows:
	// author:"John Smith" AND author:"Jane Smith"

Should we support and how do we represent?
For example:

 author:"John Smith" author:"Jane Smith"
This functionally is the same as using the OR Boolean operator, as follows:
author:"John Smith" OR author:"Jane Smith

Is NOT author:"John Smith" NOT author:"Jane Smith" became NOT author:"John Smith" OR NOT author:"Jane Smith ?

author:"John Smith" filetype:docx
This is the same as using the AND Boolean operator, as follows:
author:"John Smith" AND filetype:docx

Is NOT author:"John Smith" NOT filetype:docx" became NOT author:"John Smith" AND NOT filetype:docx ?

[fschade]

author

the ms spec defines when to use a implicit AND and when to use a implicit OR, but negotiations aren't mentioned.

my 5 cents: since no description is available for those cases i would say, as soon as someone uses an operator as edge between two nodes:

┌─────────────┐                            ┌───────────────┐
│             │        ┌─────────┐         │               │
│QUERY-NODE-1 ├────────┤ OP-EDGE ├─────────┤ QUERY-NODE-2  │
│             │        └─────────┘         │               │
└─────────────┘                            └───────────────┘

we do not add any other implicit type there. cC.: @tbsbdr

[kobergj]

Looks fluffy 👍 Just one question and one typo.

I assume documentation is done as a follow up?

[fschade]

I assume documentation is done as a follow up?

🤣 SURE, to be more exact, KQL is not enabled at the moment, but in the follow up we will enable it and provide a readme and with the help of @mmattel a even more fluffy docs page 💋

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

68.4% Coverage
0.0% Duplication