Migration to Elasticsearch

[josegar74]

The move from Lucene library to a on the shelf search engine like Solr or Elasticsearch is mainly motivated to:

• first improve performances on large catalogue and allowing clustering,
• then solve a number of limitations related to current Lucene implementation which would require major efforts.

Moving to ElasticSearch will bring a lot of flexibility on configuring the indexing and search in the catalogue. This page highlight the main benefits of this move and the current progress. GeoNetwork developers are looking for funding to continue this task.

Current limitations

• GeoNetwork Lucene index can not be shared across catalogues making clustering not feasible (unless some synchronization mechanism is set up).
• Searches
  • Limited set of operators (eg. not operator, phrase search)
  • Issue with " character used in search
  • Suggestions may display private records information (even if records by themselves are not visible)
  • Facets need to be configured to be available
  • Limited boosting mechanism
• Search results are complex to customize (needs indexing, dump fields, restart and UI change)
• Multilingual support implementation is currently quite complex (and could be simplified)

Use cases to illustrate some of the benefits

Main benefits that this move can bring to the application concern:

• Searching:
  • Better & more relevant searches (eg. scoring & boosting)
  • Combined search on features (eg. indexed from WFS sources) and records
  • Analysis: find similar documents, facet on any fields
  • Search response can use objects instead of current text delimited (eg. link, contact)

• Client can request the fields required and as such limit response size
• Indexing:
  • Synonyms supports (eg. INSPIRE themes Elevation correspond to Annex II)
  • Advanced analysis & filtering chain
  • Simplify multilingual support

This change will also probably improve search and indexing performance and still support spatial searches.

It also target to simplify the codebase and as such will make the life easier to new comers:

The following sections illustrates some of the benefits:

More flexible facets (named aggregations)

Elasticsearch API allows to:

• create aggregations on all fields with sort, limit, paging
• ranges (for dates or numerics)

• nested aggregations can be defined so we can create tree structure eg. Level 1: By resources types > Level 2: By status
• not only count of documents, aggregations can be used for analytics (eg. sum, avg)
• spatial aggregations can be used to create heatmaps
• matrix
• ...

Better suggestions

Current suggestion does not always take user privileges in account. Elasticsearch allows to use suggesters on any fields indexed (and combining searches)

eg.

More to analyze:

• Suggest terms similar too
• Phrase suggestion

More like this

"More like this" provides suggestion on similar record to the one you're currently viewing. Similarity could be define based on which fields the similarity is computed and the frequency of terms (needs more testing on how to define more like this parameters)

Dynamic dashboards

Once indexed in ElasticSearch, Kibana can be used to analyze the content of your catalogue. Kibana can be used to make analysis on the catalogue in order to promote your catalogue on CMS or third party website and can also be used to improve the content of records by searching invalid values.

• Existing dashboard support - see Elasticsearch / indexing for dashboard, search stats & healthchecks #2082

• Thematic portal embedding dashboards based on metadata analysis http://www.emodnet-atlantic.eu/Challenges/Wind-Farm-Siting

Dashboard can also focus on geographical extents:

Performances

After basic testing with ab it looks like we could expect to be 5 time faster than current services for search/facets/indexing.

Migration roadmap

Dev branch is available here https://github.com/geonetwork/core-geonetwork/tree/es

This migration is a major work and will require iterations in order to cover the full scope of what GeoNetwork search & indexing related features actually covers. The process of migration is described as a succession of levels from Level 0 providing the minimal set of features to the last Level which could provide the same level of features. Therefore this migration can also be an opportunity to remove unused features and simplify the codebase.

Level 0 (Proof of concept)

Level 0 means that the application starts, create the index if not existing, index document from the database and the main search is available. This is the proof of concept allowing to analyze what will be the implementation main goals.

The search API is provided by the /api/search/records/_search service:

Tasks for this level are:

• Removing Lucene dependency. This is required to add Elasticsearch as a dependency which depends on a much more recent version of Lucene.
• Indexing support for 19139/19110/Dublin core
• Add an index proxy taking care of forwarding search from the user interface to the index. The proxy also takes care of:
  • Add user privileges to search queries
  • Add user info to search response
  • Turn off current proxy
• Do not fail on startup if ES is not available. Which means that GeoNetwork can start without a running Elasticsearch and init the index once Elasticsearch is available.
• UI / Main search form / Use new search service
  • Define draft core response structure
  • Simple one level facet support
  • Full text search support
• Define index fields structures, naming and conventions
• More like this concept

• Suggestion and phrase search concept

• Admin console / Add status information on the index
• Admin console / Add tools required to manage the new index

Funding: This level 0 of features was developed mainly during the 2018 Bolsena codesprint.

Level 1

Level 1 target a fully working user interface based on the new search service. Search is used in many places from the home page to the associated resources panel in the editor.

Technical challenges:

• Request only the required fields
• One level aggregations keeping in mind that nested aggregation will be implemented in the future

Tasks for this level are:

• Application can start even if index is down.

• Search / wire all the UI (home/edit/admin) to the new search service with proper search results

• Search / Return in _source only the field required by the UI (performance)

• Translation of codelist

• Facets / restore full support of one level facet with configuration from the settings

  • Configuration is now available in the admin per module ie. search/editor board

• Add support to filter aggregations

GET /gn-records/_search
{
  "aggs": {
    "messages" : {
      "filters" : {
        "filters" : {
          "availableInViewService" :  {
              "query_string": {
                "query": "+linkProtocol:/OGC:WMS.*/"
              }
            },
          "availableInDownloadService" :  {
              "query_string": {
                "query": "+linkProtocol:/OGC:WFS.*/"
              }
            }
        }
      }
    }},
  "query": {
  "match_all": {}
}}

• Utility / XSLUtils / get index fields value
• Search related
• Cleaning / Remove unused Lucene XSLT from schema plugins
• Autocompletion and UI configuration

Various Elasticsearch queries can be used to configure autocompletion.

By default, a multi_match on anytext + its ngram associated fields is configured in order to propose record titles based on analysis of partial word match.

Funding: This level 1 of features was developed mainly:

• during the 2019 Bolsena codesprint
• July codesprint (https://github.com/geonetwork/core-geonetwork/wiki/Elasticsearch-migration-July-2019-codesprint) sponsored by EEA
• titellus

Level 2

Level 2 focus on restoring CSW and improving aggregations (aka facet) supports.

Tasks for this level are:

• CSW support

• CSW / support geometries with And / Or conditions.

• Virtual CSW (deprecated and replace by portal - which include a virtual CSW)

• Migrate to RestHighLevelClient Java client API instead of JEST library (used because no dependency on Lucene)

• Selection / Restore selection manager

• Selection / Restore MEF export

• Selection / Restore PDF/CSV export

• How to integrate ES for running tests and to package the installer

• Improve index status checker by reporting status

• create automatically index when it does exist

• Multi portal support / Restore portal filter injection

• Facet / Allow OR in a category

• Facet / Allow not a value eg. not a service

• Hierarchical facet eg. GEMET. Hierarchy of facet is now supported using 2 approaches:

  • sub aggregation concept in Elasticsearch which allows to have nested aggregations eg. below on resource type > format
  • path hierarchy using a separator eg. below on GEMET thesaurus

At the end of level 2, GeoNetwork should provide main functionalities for users requiring search (including CSW)/editing/map/admin console.

• Facet / Load more values

• Histogram aggregation support

Funding: This level 2 of features was developed mainly:

• December codesprint https://github.com/geonetwork/core-geonetwork/wiki/Openlayers-migration-codesprint-December-2019#elasticsearch-migration
• titellus
• Metawal
• EEA

Level 3

• Search suggestion / Configuration (which fields to suggest on)
• Index / Subtemplates
• Map / wire all searches
• Index / Handle graphic overview with or without label
• Index / Add Anchor support
• Search / Support special character eg. search for "1+1=2"

• Admin / Harvester / List records for current harvester

Funding:

• titellus
• Metawal

Level 4

Level 4 focused on making a first beta release of GeoNetwork on Elasticsearch that can be used for real.

• Docker composition
• Documentation for installation
• Kibana default dashboards
• Testing & listing missing functionalities (define roadmap for next levels)
• ES / Index / Temporal coverage using date range type (see https://www.elastic.co/guide/en/elasticsearch/reference/current/range.html)

• ES / suggestion / current filter apply
• ES / sort / Most recent (to easily find latest created)
• ES / Index / Temporal coverage using date range type
• Release a GeoNetwork 3.99.0 ?

Funding:

• EEA
• Metawal

Reference documents:

• Solr POC https://github.com/geonetwork/core-geonetwork/compare/3.2.x...titellus:solr?expand=1

[fxprunayre]

CFV https://sourceforge.net/p/geonetwork/mailman/message/36995405/

• +1 Jose, Florent, Emanuele, Jo, Jeroen, Francois

Actions:

• Version changed to 4.0.0.alpha-1

• Branch es will be renamed to 4.0.x https://github.com/geonetwork/core-geonetwork/tree/4.0.x

• Tag created with 4.0.0.alpha-1 https://github.com/geonetwork/core-geonetwork/tree/4.0.0-alpha.1

• Release on sourceforge https://sourceforge.net/projects/geonetwork/files/GeoNetwork_unstable_development_versions/

• Publish the documentation for this release on the website (https://github.com/geonetwork/doc/tree/4.0.x.) manuals/4.0.x/en/index.html

• Website update Annouce 4.0.0-alpha.1 website#21

• Website publilcation

• Publish the docker version in GeoNetwork organization

• Create a nightly build in http://nightlybuild.geonetwork-opensource.org/

Note:

• db migration between alpha releases will not be created (db changes should be minor so only a migration from 3.10.x to 4.0.0 will be created)

[fxprunayre]

Closing, this is now https://github.com/geonetwork/core-geonetwork/tree/4.0.x