Support search qualifiers

[davidsvantesson]

• Gitea version (or commit ref): 1.10.0+dev-375-g8a828500e

Description

Gitea should support search qualifiers when searching for repositories, issues, PRs or users. This would allow more flexible options when searching.

Example when searching for repos:
topic:
is:private
is:public
owner:name

Search qualifiers shall always be AND search terms (in contrast to text search which is OR).

Note: Gitea has divided search terms by comma and not space so this example:
"sentence of words,topic:mytopic,separate sentance"
would search for repositories where "sentence of words" OR "separate sentance" is in the name or description AND has the topic "mytopic".

Maybe this can be solved by indexing with bleve and using required and exclusion of fields:
https://blevesearch.com/docs/Query-String-Query/

[guillep2k]

* Gitea version (or commit ref): master

Gitea version (or commit ref): 1.10.0+dev-375-g8a828500e

You can use the footer from try.gitea.io. I know this is nit-picky of me, but master is a relative term. 😁

[davidsvantesson]

Sure, I am usefully more careful when posting bugs.

[davidsvantesson]

Another benefit of this way of searching is that it is more clear for the user when selecting search options in the UI. For example if selecting "Assigned to you" it would end up as "assignee:davidsvantesson" in the search field.

[guillep2k]

Perhaps we should consider using a localizable resource for this (e.g. "assignee", "asignado", "asignato", etc.). Preferrably with many options. For example, in app.ini:

[search.qualifiers]
ASSIGNEE = assignee, asignado

To avoid complicating the code too much, the system could just use the first one in the list when building the search string for the search field in the returned form, like some sort of "normalization".

As for other kinds of keywords (like "fixes", "closes", etc.) this is not something you want to change whenever you update the translations from Crowding.

[davidsvantesson]

I am so used working in English so I didn't consider localization, but it sounds like a good idea.

I also think the current behavior with splitting on "," should be reconsidered, although it would be a breaking change. It doesn't feel very standardized and is not documented (I didn't know about it before looking into code). I think it is better to search each white-space separated word separately and allow quotation marks to search for an exact match. Then package text/scanner can be used with default settings to split the string.

[guillep2k]

I am so used working in English so I didn't consider localization, but it sounds like a good idea.

I also think the current behavior with splitting on "," should be reconsidered, although it would be a breaking change. It doesn't feel very standardized and is not documented (I didn't know about it before looking into code). I think it is better to search each white-space separated word separately and allow quotation marks to search for an exact match. Then package text/scanner can be used with default settings to split the string.

I agree with you about the spaces. As a breaking change, there are three options the way I see this:

1. Change the behavior, make the users aware and get used to it (Github and search engines works with spaces, so I think this won't get too much resistance).
2. Keep the current behavior.
3. Add an app.ini flag to switch between old and new behaviors.

In this matter I'm all for (1), but it's just my opinion. If implementing (3) is trivial, it may be added and decide a sensible default for it by consensus.

[bagasme]

For 1.10, I choose option 3 (as a transition into proposed behavior), and on version 1.11, use option 1 (remove transitional flag).

[davidsvantesson]

@bagasme This is only a proposal issue so far.

I think the only change of behavior would be that comma should be replaced with no comma, and exact searches need quotes. The previous behavior was undocumented anyhow, I think we can add documentation for the new behavior. The special queries can be kept for backward compatibility (like &topic=1)

Preferably there shall be a "best match" sorting, like how many of the search words occurs in the repo. I don't know how hard it is to make an effective algorithm of that.

[bagasme]

@davidsvantesson Besides documenting (proposed) new behavior, the old one should also be documented too. This come handy when we switch to new behavior, and users complain when their old/undocumented syntax doesn't work anymore and they want explanations...

[davidsvantesson]

I tried to do some research on 'best match' text searches for sql. Most solutions are tied to specific sql databases (eg. oracle: REGEXP_COUNT, MsSQL: Rank).
I found one solution with standard sql to count how many of the words occur for each row. It could maybe work to order by as a best match.

[guillep2k]

Preferably there shall be a "best match" sorting, like how many of the search words occurs in the repo. I don't know how hard it is to make an effective algorithm of that.

I think bleve can do that; it's the default text search engine for issues. I don't mind if simpler SQL indexing lacks this kind of feature.

For SQL search it's hard to decide what should count as "best match". Number of times a word appears in the title? (that should be low) Number of times it appears in the body? Number of comments, counting the body, that contain the word? Most of these will be very heavy on the database. Hence, bleve should be preferred.

A "best match" is more useful when you do some semantic analysis on the text, like counting any of "do", "did", "done" as synonyms for each other. And that's language dependent; we could make Gitea support x number of languages, but that's another whole can of worms.

[davidsvantesson]

Sounds reasonable. So a best match search for repositories would need a bleve only for name, description and other repository metadata.
My concern is that if adding more search words only add more repositories being matched without any reasonable sorting, it might not be very useful. Usually you want to narrow down your search by adding more search terms.

[davidsvantesson]

I have not understood bleve fully, but it seems it can support this directly if just indexing different fields of the repo metadata:
https://blevesearch.com/docs/Query-String-Query/

[guillep2k]

We're not using that interface. We're one level below querying directly constructing the objects manually:

gitea/modules/indexer/issues/bleve.go

Lines 222 to 231 in 6551a9d

	indexerQuery := bleve.NewConjunctionQuery(
	numericEqualityQuery(repoID, "RepoID"),
	bleve.NewDisjunctionQuery(
	newMatchPhraseQuery(keyword, "Title", issueIndexerAnalyzer),
	newMatchPhraseQuery(keyword, "Content", issueIndexerAnalyzer),
	newMatchPhraseQuery(keyword, "Comments", issueIndexerAnalyzer),
	))
	search := bleve.NewSearchRequestOptions(indexerQuery, limit, start, false)
	result, err := b.indexer.Search(search)

But of course that means we can do it either way.

[guillep2k]

Here's the code where the analyzers are decided:

gitea/modules/indexer/issues/bleve.go

Lines 129 to 133 in 6551a9d

	} else if err = mapping.AddCustomAnalyzer(issueIndexerAnalyzer, map[string]interface{}{
	"type": custom.Name,
	"char_filters": []string{},
	"tokenizer": unicode.Name,
	"token_filters": []string{unicodeNormalizeName, lowercase.Name},

Those decide what kind of analysis you want on the strings (they must be decided at the moment the index is built!).