Add support for Lucene as search syntax #8206

koppor · 2021-11-02T00:29:24Z

Fixes #1975

As far as I tested, the search syntax of groups can be reused (= signs also work at Lucene).

The search using regular expressions is different in Lucene. One has to use /.../ to indicate regular expressions. Thus, groups can now also use regular expressions.

Left TODOs:

The reg ex switch in the search bar should be removed. All reg ex searches should be handled by the lucene-based search
The search rule describer for Lucene has to be created. Currently only a dummy.
Think of removing the search grammar completely
Optional: Implement Glob patterns - first idea: http://svn.apache.org/repos/asf/jakarta/oro/trunk/src/java/org/apache/oro/text/ --> GlobCompiler.java -- or "just" transform the string to a regex --> https://stackoverflow.com/q/1247772/873282

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

btut

Nitpick and the static checker found an unused import.

CHANGELOG.md

btut · 2021-11-03T13:28:23Z

I think overall it would make sense to rework the search mechanism to use a Lucene index to improve speed.

A few thoughts on that:

Currently, search is done for each entry individually. So a query is executed on each entry and a boolean return value indicates whether the entry fits the query or not. With Lucene, a query is executed on the index and returns entries.
For the Fulltext-search, I had to implement a workaround, storing all returned entries that fit a querry and then, when the query is executed on an entry, it actually checks if the entry is in that list. I dislike that workaround.
Lucene does not provide a binary decision whether the entry matches the query. Lucene provides a score.
This means it cannot (should not) be used as a binary filter over the entry-table (as is done now). Instead, the entry-table should be sorted according to the scoring. To give the user feedback on where the query was found in an entry, it would make sense to leverage Lucene's...
Highlighting. Lucene can generate html formatted (converted to a JavaFX TextFlow for the fulltext results) highlighted string, similar to what google does when displaying search results.

calixtus · 2021-11-03T21:55:26Z

I sense a damn f**** cool possible new (or renewed) feature for JabRef here. Sort by lucene score and/or highlight/opaque entries by score - like floating results in ancient swing times.

btut · 2021-11-04T09:27:30Z

I agree. If done well, this could be a very neat feature and the speed-up would be icing on the cake.

tobiasdiez

I'm also in favor of completely removing our existing search infrastructure and migrate completely to Lucene.
Not sure whats more efficient: first only except Lucene syntax in the search query, and in a second PR throw out the old stuff; or rewrite everything in one go.

src/main/java/org/jabref/model/search/rules/ContainsBasedSearchRule.java

CHANGELOG.md

koppor · 2021-11-14T19:15:47Z

Current blocker - Lucene does not support == and !=.

= inexact match. In Lucene, a glob pattern would be needed. In the current implementation, I did not enforce that, but implemented an inexact match
== is an exact match. Parsing error in Lucene
!= is NOT. Parsing error in Lucene

How to proceed?

Keep our search (and maybe map to Lucene for full text support)
Drop support for == and !=

tobiasdiez · 2021-11-14T22:04:58Z

I would completely drop our search syntax and only use lucenes one. If users really want an equals instead of contains search, then we can add this later (for example by adding a field alias with no tokenizer).

koppor · 2021-11-15T05:38:18Z

Note to self: Read Lucene documentation thoroughly --> http://www.lucenetutorial.com/lucene-query-syntax.html

What is a match?

Currently: title:Jab --> Matches "JabRef tool"
Lucene: title:Jab --> does not match "JabRef tool"

Currently: title:Jab --> Matches "JabRef tool"
Lucene: title:Jab* --> Matches "JabRef tool"

Currently: title:JabRef --> Matches "JabRef tool"
Lucene: title:JabRef --> Matches "JabRef tool"

It is OK for me to implement like that. The "only" concern I have is that it does not match the UX one has from WWW search enginges (Google, Bing, ...)

btut · 2021-11-15T06:40:42Z

If users really want an equals instead of contains search, then we can add this later (for example by adding a field alias with no tokenizer).

I think the most efficient way would be to do a fuzzy search anyhow and then do the equals search on the results of the fuzzy search as we did before. I think that would be better than storing everything twice (tokenized and untokenized).

... After writing this I did a short google search and it appears like exact matches are possible in lucene using quotes. https://stackoverflow.com/questions/37495639/how-to-match-exact-text-in-lucene-search
Isn't that exactly what we want?

koppor · 2022-04-04T22:35:23Z

Minor comment: I rebased this PR on main, because we merged #8636

koppor · 2022-04-19T20:25:01Z

ADR https://github.com/JabRef/jabref/blob/main/docs/adr/0015-support-an-abstract-query-syntax-for-query-conversion.md should be updated

# Conflicts: # src/main/java/org/jabref/logic/search/rules/ContainsBasedSearchRule.java # src/main/java/org/jabref/logic/search/rules/GrammarBasedSearchRule.java # src/main/java/org/jabref/logic/search/rules/RegexBasedSearchRule.java

github-actions · 2022-07-07T21:47:34Z

src/test/java/org/jabref/logic/search/rules/LuceneBasedSearchRuleTest.java

+    public void findsCaseInSensitive(String query) {
+        assertTrue(luceneBasedSearchRuleCaseInsensitive.validateSearchStrings(query));
+        assertTrue(luceneBasedSearchRuleCaseInsensitive.applyRule(query, bibEntry));
+    }


🚫 [reviewdog] <com.puppycrawl.tools.checkstyle.checks.regexp.RegexpMultilineCheck> _{reported by reviewdog 🐶}
Blank line at end of block should be removed

github-actions · 2022-07-07T21:47:34Z

src/test/java/org/jabref/logic/search/rules/SearchRulesTest.java

+    void simpleFieldedLuceneQueryReturnsLuceneBasedSearchRule() {
+        SearchRule searchRule = SearchRules.getSearchRuleByQuery("title:test", EnumSet.noneOf(SearchRules.SearchFlags.class));
+        assertInstanceOf(LuceneBasedSearchRule.class, searchRule);
+    }


🚫 [reviewdog] <com.puppycrawl.tools.checkstyle.checks.regexp.RegexpMultilineCheck> _{reported by reviewdog 🐶}
Blank line at end of block should be removed

koppor · 2022-07-07T21:57:03Z

... After writing this I did a short google search and it appears like exact matches are possible in lucene using quotes.

I would object - based on https://www.lucenetutorial.com/lucene-query-syntax.html

I would follow the suggestion of #8206 (comment) and implement the interpretation of the Lucene syntax "more relaxed".

ThiloteE · 2022-07-08T12:37:08Z

Question: getting rid of current search syntax will break groups that are configured to search based on

searching for a keyword
free search expression

right? Is there a way the syntax could be backwards compatible?

btut · 2022-07-09T06:55:03Z

Question: getting rid of current search syntax will break groups that are configured to search based on

searching for a keyword

free search expression

right? Is there a way the syntax could be backwards compatible?

I never used that feature so I had to look it up first.
In general, one goal of the lucene-implementation would be to get rid of all the old search infrastructure. This is hard to do when keeping backwards compatibility. But there is hope. 'Simple' searches, just looking for a string in a field, should straight up just work in lucene as well. The problems arise when we look at the case-sensitive and regex switches and to scoring:

Lucene does not support case-insensitive searches. Searches are always case-sensitive. If you want a case-insensitive search, you lowercase everything that you put into the index and your search strings.
I do not know if the regex implementation of JabRef and Lucene match. If so, great. If not, we could have a 'migration' feature that parses search strings in the group settings on startup and if they match JabRef syntax they are translated to lucene syntax. We need the regex parser for other things anyways so we wouldn't get rid of it anyways.
The problem that makes the current fulltext-search less usable than it could be is the scoring. Lucene almost always finds something. It might not be a good fit but still better than other entries. We need a good scoring mechanism to filter out irrelevant results. I don't know if a simple score-threshold would suffice or we need something more sophisticated.

koppor · 2022-09-26T19:14:39Z

Superseeded by #8963

This PR can be used to check for similarities and differences by using the Lucene engine or (our) custom search.

calixtus mentioned this pull request Nov 2, 2021

Implement ChipView for the search JabRef/jabref-koppor#341

Open

koppor mentioned this pull request Nov 3, 2021

Create disjoint subgroups #7049

Open

koppor linked an issue Nov 3, 2021 that may be closed by this pull request

Create disjoint subgroups #7049

Open

koppor removed a link to an issue Nov 3, 2021

Create disjoint subgroups #7049

Open

btut suggested changes Nov 3, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

tobiasdiez requested changes Nov 4, 2021

View reviewed changes

src/main/java/org/jabref/model/search/rules/ContainsBasedSearchRule.java Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

koppor mentioned this pull request Nov 14, 2021

Update search syntax JabRef/user-documentation#373

Closed

This was referenced Nov 14, 2021

Fix search: NOT binds more than AND #8241

Merged

Add doc tests #8242

Merged

koppor added the component: search label Nov 14, 2021

koppor mentioned this pull request Nov 20, 2021

Technical Documentation and Prototype josphstar/jabref#2

Open

josphstar mentioned this pull request Dec 22, 2021

Prototype2.0 josphstar/jabref#7

Merged

dmadoery mentioned this pull request Dec 22, 2021

Prototype Implement better search user interface #341 #8356

Open

5 tasks

julianbopp mentioned this pull request Dec 22, 2021

Implement better search issue#341 josphstar/jabref#9

Closed

koppor added this to the v6.0 milestone Dec 23, 2021

koppor added the project: gsoc label Mar 26, 2022

tobiasdiez mentioned this pull request Mar 27, 2022

Improve search interface #7423

Open

ThiloteE mentioned this pull request Mar 30, 2022

Incomplete fulltext search results #8626

Closed

2 tasks

koppor mentioned this pull request Apr 3, 2022

Refine search code #8636

Merged

6 tasks

koppor added 4 commits April 5, 2022 00:24

Add support for Lucene-based search

2df81db

Add support for grouping

b9b3a8f

Refine CHANGELOG.md

d118d59

Move search from model to logic

de493cd

koppor and others added 4 commits April 5, 2022 00:29

Add support for "anyfield", "anykeyword"

8d685dc

Add test case mirroring our documentation

3202fd0

Fix checkstyle

6a165b0

Fix missing imports

9c8e639

koppor force-pushed the refine-search branch from aef64af to 9c8e639 Compare April 4, 2022 22:33

AEgit mentioned this pull request Apr 29, 2022

Issues with search and groups and associated feature request (floating mode) #4237

Closed

ThiloteE mentioned this pull request May 25, 2022

Search for an entry without triggering group or other fields #7996

Closed

AEgit mentioned this pull request May 30, 2022

Unexpected behaviour: Groups search - Item assignment #3176

Closed

koppor mentioned this pull request Jun 13, 2022

Explain search better JabRef/jabref-koppor#214

Open

github-actions bot reviewed Jul 7, 2022

View reviewed changes

koppor added 2 commits July 7, 2022 23:52

Add some comments

9ab3ae6

Fix checkstyle

59c0d8f

btut removed the project: gsoc label Jul 27, 2022

koppor closed this Sep 26, 2022

koppor deleted the refine-search branch September 26, 2022 19:14

LoayGhreeb mentioned this pull request Sep 2, 2024

Lucene search #11542

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Lucene as search syntax #8206

Add support for Lucene as search syntax #8206

koppor commented Nov 2, 2021 •

edited

Loading

btut left a comment

btut commented Nov 3, 2021

calixtus commented Nov 3, 2021

btut commented Nov 4, 2021

tobiasdiez left a comment

koppor commented Nov 14, 2021

tobiasdiez commented Nov 14, 2021

koppor commented Nov 15, 2021

btut commented Nov 15, 2021

koppor commented Apr 4, 2022

koppor commented Apr 19, 2022

github-actions bot Jul 7, 2022

github-actions bot Jul 7, 2022

koppor commented Jul 7, 2022

ThiloteE commented Jul 8, 2022

btut commented Jul 9, 2022

koppor commented Sep 26, 2022

Add support for Lucene as search syntax #8206

Add support for Lucene as search syntax #8206

Conversation

koppor commented Nov 2, 2021 • edited Loading

btut left a comment

Choose a reason for hiding this comment

btut commented Nov 3, 2021

calixtus commented Nov 3, 2021

btut commented Nov 4, 2021

tobiasdiez left a comment

Choose a reason for hiding this comment

koppor commented Nov 14, 2021

tobiasdiez commented Nov 14, 2021

koppor commented Nov 15, 2021

btut commented Nov 15, 2021

koppor commented Apr 4, 2022

koppor commented Apr 19, 2022

github-actions bot Jul 7, 2022

Choose a reason for hiding this comment

github-actions bot Jul 7, 2022

Choose a reason for hiding this comment

koppor commented Jul 7, 2022

ThiloteE commented Jul 8, 2022

btut commented Jul 9, 2022

koppor commented Sep 26, 2022

koppor commented Nov 2, 2021 •

edited

Loading