Replace native ElasticSearch integration with Hibernate Search

[matthias-ronge]

This development aims to replace the native ElasticSearch integration with the Hibernate Search framework, which uses ElasticSearch. Little shall change for those starting the application out.

The exchange should take place in two steps:

1. A version is created in which the existing ElasticSearch integration has been removed. This requires the following steps:

• 1a) remove test for indexing
• 1b) remove indexing page
• 1c) remove ElasticSearch usage (remove DTO objects, unify saveToDatabase and saveToIndex in save function)
• 1d) remove database column indexAction from all tables and objects

Goal: The application runs alone with the database. Only the filters and advanced search may not work.

2. A version is created in which Hibernate Search has been implemented. This requires the following steps:

• 2a) Insert Hibernate Search framework. Annotate table columns.
• 2b) Implement bulk indexing (reimplement indexing page)
• 2c) Implement / customize search. The filters are working again.
• 2d) The advanced search works again.
• 2e) Search in non-database data (METS files) in a meaningful way
• 2f) optimization of the search

Goal is to fix a lot of errors and problems with indexing, see #5546.

This development, funded by the Kitodo e.V. Development Fund

[stweil]

This development aims to replace the native ElasticSearch integration with the Hibernate Search framework, which uses ElasticSearch. Little shall change for those starting the application out.

Isn't there consense that Elasticsearch shall be replaced by OpenSearch? Then the text should be "[...] which uses OpenSearch".

[henning-gerhardt]

This development aims to replace the native ElasticSearch integration with the Hibernate Search framework, which uses ElasticSearch. Little shall change for those starting the application out.

Isn't there consense that Elasticsearch shall be replaced by OpenSearch? Then the text should be "[...] which uses OpenSearch".

There is only the consense to use ElasticSearch or OpenSearch with HibernateSearch.

[stweil]

Then the text should be "[...] which works with either Elasticsearch or OpenSearch".

@matthias-ronge, what is the status of this issue and the whole work on support for HibernateSearch?

[solth]

@stweil We are in the middle of the review. If you want to support us with the review, you are very welcome to do so.

[BartChris]

This development aims to replace the native ElasticSearch integration with the Hibernate Search framework, which uses ElasticSearch. Little shall change for those starting the application out.

Isn't there consense that Elasticsearch shall be replaced by OpenSearch? Then the text should be "[...] which uses OpenSearch".

There is only the consense to use ElasticSearch or OpenSearch with HibernateSearch.

Maybe relevant here: https://www.elastic.co/de/blog/elasticsearch-is-open-source-again

[stweil]

Maybe relevant here: https://www.elastic.co/de/blog/elasticsearch-is-open-source-again

The new Elasticsearch releases use AGPL, so I am afraid that they cannot be used with Kitodo.Production which uses GPL – unless Kitodo also switches to AGPL.